Image processing method and image processing apparatus

ABSTRACT

An image processing method with which occurrence of an error is suppressed is an image processing method for processing an image signal generated by encoding a plurality of pictures. The method includes: obtaining, from the image signal, a parameter indicating restriction on a reference relation between at least one of the plurality of pictures and an other one of the plurality of pictures in the encoding; and performing restriction alleviation processing for alleviating the restriction on the reference relation indicated by the parameter.

CROSS REFERENCE TO RELATED APPLICATION

The present application claims the benefit of Japanese PatentApplication No. 2013-099001 filed on May 9, 2013 and Japanese PatentApplication No. 2013-099002 filed on May 9, 2013. The entire disclosuresof the above-identified applications, including the specifications,drawings and claims are incorporated herein by reference in thereentirety.

FIELD

The present disclosure relates to an image processing method and animage processing apparatus for processing an image signal.

BACKGROUND

In order to compress audio data and video data, the audio codingstandards and the video coding standards have been developed. Theexamples of the video coding standards include the ITU-T standardsdenoted as H. 26x, and the ISO/IEC standards denoted as MPEG-x (see NonPatent Literature 1, for example). The latest video coding standard isthe standard denoted as H.264/MPEG-4AVC. In addition, a next-generationcoding standard denoted as High Efficiency Video Coding (HEVC) has beenunder consideration in recent years (see Non Patent Literature 2).

CITATION LIST Non Patent Literature Non Patent Literature 1

-   ISO/IEC 14496-10 “MPEG-4 Part 10 Advanced Video Coding”

Non Patent Literature 2

-   Joint Collaborative Team on Video Coding (JCT-VC) of ITU-T SG16 WP3    and ISO/IEC JTC1/SC29/WG11 12th Meeting: Geneva, SW, -1-10 Jan.    2013, JCTVC-L1003, “High Efficiency Video Coding (HEVC) text    specification draft10”,    http://phenix.int-evry.fr/jct/doc_end_user/documents/12_Geneva/wg11/JCTVC-L1003-v34.zip

SUMMARY Technical Problem

However, the image processing methods of the above-described NPL 1 andNPL 2 pose a problem that they are prone to an error.

In view of the above, one non-limiting and exemplary embodiment providesan image processing method with which occurrence of an error issuppressed.

Solution to Problem

An image processing method according to an aspect of the presentdisclosure is an image processing method for processing an image signalgenerated by encoding a plurality of pictures, the image processingmethod including: obtaining a parameter from the image signal, theparameter indicating restriction on a reference relation between atleast one of the plurality of pictures and an other one of the pluralityof pictures in the encoding; and performing restriction alleviationprocessing for alleviating the restriction on the reference relationindicated by the parameter.

General and specific aspects disclosed above may be implemented using asystem, a method, an integrated circuit, a computer program, or acomputer-readable recording medium such as a CD-ROM, or any combinationof systems, methods, integrated circuits, computer programs, orcomputer-readable recording media.

Additional benefits and advantages of the disclosed embodiments will beapparent from the Specification and Drawings. The benefits and/oradvantages may be individually obtained by the various embodiments andfeatures of the Specification and Drawings, which need not all beprovided in order to obtain one or more of such benefits and/oradvantages.

Advantageous Effects

An image processing method disclosed herein enables suppressingoccurrence of an error.

BRIEF DESCRIPTION OF DRAWINGS

These and other objects, advantages and features of the disclosure willbecome apparent from the following description thereof taken inconjunction with the accompanying drawings that illustrate a specificembodiment of the present disclosure.

FIG. 1 is a diagram which illustrates a relation between a random accesspoint (RAP) picture and a leading picture.

FIG. 2 is a diagram for explaining the leading picture when the RAPpicture is an IDR picture.

FIG. 3 is a diagram for explaining the leading picture when the RAPpicture is a BLA picture.

FIG. 4 is a diagram for explaining classification of the leadingpicture.

FIG. 5 is a diagram which illustrates NAL unit types related to the RAPpicture and the leading picture.

FIG. 6 is a flowchart which illustrates an operation of a decoder whichis an image processing apparatus according to Embodiment 1.

FIG. 7 is a block diagram which illustrates a configuration of the imageprocessing apparatus which is a decoder according to Embodiment 1.

FIG. 8 is a flowchart which illustrates an operation of a NAL unitsyntax analysis unit of the decoder according to Embodiment 1.

FIG. 9 is a flowchart which illustrates an operation of the NAL unitsyntax analysis unit of the decoder according to Embodiment 2.

FIG. 10 is a flowchart which illustrates an operation of the decoderaccording to a modification example of Embodiment 2.

FIG. 11 is a diagram which illustrates the NAL unit types used inEmbodiment 3.

FIG. 12 is a diagram for explaining scalable decode used in Embodiment3.

FIG. 13 is a diagram for explaining a TSA picture used in Embodiment 3.

FIG. 14 is a diagram for explaining a STSA picture used in Embodiment 3.

FIG. 15 is a flowchart which illustrates an example of decodingprocessing performed by a decoder which is the image processingapparatus according to Embodiment 3.

FIG. 16 is a flowchart which illustrates an example of decodingprocessing performed by a decoder which is the image processingapparatus according to Embodiment 3.

FIG. 17A is a diagram for explaining a temporal nesting flag used inEmbodiment 3.

FIG. 17B is a diagram for explaining the temporal nesting flag used inEmbodiment 3.

FIG. 17C is a diagram for explaining a temporal nesting flag used inEmbodiment 3.

FIG. 17D is a diagram for explaining a temporal nesting flag used inEmbodiment 3.

FIG. 18 is a flowchart which illustrates another example of decodingprocessing performed by the decoder which is the image processingapparatus according to Embodiment 3.

FIG. 19A is a flowchart which illustrates an image processing methodaccording to an aspect of the present disclosure.

FIG. 19B is a block diagram which illustrates a configuration of theimage processing apparatus according to an aspect of the presentdisclosure.

FIG. 20 shows an overall configuration of a content providing system forimplementing content distribution services.

FIG. 21 shows an overall configuration of a digital broadcasting system.

FIG. 22 shows a block diagram illustrating an example of a configurationof a television.

FIG. 23 shows a block diagram illustrating an example of a configurationof an information reproducing/recording unit that reads and writesinformation from and on a recording medium that is an optical disk.

FIG. 24 shows an example of a configuration of a recording medium thatis an optical disk.

FIG. 25A shows an example of a cellular phone.

FIG. 25B is a block diagram showing an example of a configuration of acellular phone.

FIG. 26 illustrates a structure of multiplexed data.

FIG. 27 schematically shows how each stream is multiplexed inmultiplexed data.

FIG. 28 shows how a video stream is stored in a stream of PES packets inmore detail.

FIG. 29 shows a structure of TS packets and source packets in themultiplexed data.

FIG. 30 shows a data structure of a PMT.

FIG. 31 shows an internal structure of multiplexed data information.

FIG. 32 shows an internal structure of stream attribute information.

FIG. 33 shows steps for identifying video data.

FIG. 34 shows an example of a configuration of an integrated circuit forimplementing the moving picture coding method according to each ofembodiments.

FIG. 35 shows a configuration for switching between driving frequencies.

FIG. 36 shows steps for identifying video data and switching betweendriving frequencies.

FIG. 37 shows an example of a look-up table in which video datastandards are associated with driving frequencies.

FIG. 38A is a diagram showing an example of a configuration for sharinga module of a signal processing unit.

FIG. 38B is a diagram showing another example of a configuration forsharing a module of the signal processing unit.

DESCRIPTION OF EMBODIMENTS

(Underlying Knowledge Forming Basis of the Present Disclosure)

An image signal encoded using the coding technique of HEVC includes ahigher level unit for storing a video frame (picture) called a networkabstraction layer (NAL) unit.

With an apparatus or system in which an image signal is provided, it ispossible, by only decoding header information of a NAL unit, to broadlyclassify video frames which are targets for processing, without decodinga video core layer (slice layer or macroblock layer). For that reason,the header information of a NAL unit is mainly used by an application ofa system layer which lacks sufficient capacity for encoding and/ordecoding the video core layer. The header information of a NAL unitincludes a NAL unit type and a time hierarchy ID (TID), and the videocore layer includes information indicating a reference relation.

However, the reference relation indicated by the header information ofthe NAL unit contradicts an actual reference relation in the video corelayer in some cases. In such a case, there is a problem that anoperation of the system layer and a decoding operation of the video corelayer cannot be carried out.

First, the configuration of pictures included in video will be describedin detail in order to show that the above-described problem occurs.

The demand is high for reproducing recorded content (bit stream) from aposition in midstream, which is generally called as random accessreproduction. For that reason, a random access point for enabling therandom access reproduction is set in a bit stream (the above-describedimage signal) obtained as a result of encoding video.

A random access point is a point which is a starting point whenreproducing a bit stream from a position in midstream. A picture to bethe random access point is called a random access point (RAP) picture.When video is encoded using a time correlation between pictures as inMPEG-x, a bit stream obtained as a result of encoding the video cannotbe decoded from an arbitrary position. However, since the RAP picture isencoded independently of other pictures, it is possible to startdecoding from the RAP picture. The RAP picture needs to be encodedindependently of other pictures, and thus inter frame (inter picture)prediction encoding is not carried out. Only intra frame (intra picture)prediction encoding is performed on the RAP picture, and the RAP picturebecomes an intra picture (I picture).

FIG. 1 is a diagram which illustrates a relation between a random accesspoint (RAP) picture and a leading picture. A detailed description as tothe leading picture will be provided below. It is to be noted that, inFIG. 1, a rectangle frame indicates a picture, a numerical character inthe frame indicates a picture number that will be described later, and abracketed alphabet in the frame indicates a picture type (I-picture,P-picture, or B-picture).

In FIG. 1, (a) illustrates an outputting order (display order) ofuncompressed pictures which are decoded, or an inputting order of inputpictures to be encoded. The pictures are assigned numbers (picturenumbers), and pictures from picture 1 to picture 16 are illustrated asexamples of the picture numbers. It is to be noted that the picturenumbers indicate temporally preceding-following relations of input oroutput.

In FIG. 1, (b) illustrates a decoding order of encoded pictures or anencoding order of inputted pictures. More specifically, pictures areencoded or decoded in the order of picture numbers 4, 1, 2, 3, 8, 5, . .. . In other words, pictures are not encoded in the inputting orderillustrated in (a) in FIG. 1, but pictures are encoded with the orderbeing changed as illustrated in (b) in FIG. 1. Alternatively, picturesare decoded not in the outputting (display) order illustrated in (a) inFIG. 1 but in the order as illustrated in (b) in FIG. 1. Accordingly,for example, the picture 4 previously decoded is held in a buffer(decoded picture buffer (DPB)) until the time to output (display) thepicture 4 after the pictures 1, 2, and 3 are outputted (displayed).

In FIG. 1, (c) illustrates a frame reference relation used in interframe prediction encoding. A frame reference relation of pictures fromthe picture 4 which is a RAP picture to the picture 12 which is the nextRAP picture is illustrated as an example. The dotted arrow lines eachillustrate the reference between frames, indicating that a picture atthe starting point of an arrow uses a picture at the ending point of thearrow as a reference picture in prediction encoding. For example, thepictures 5, 6, and 7 which are B-pictures each refer to the picture 4and the picture 8. The picture 8 which is a P-picture refers to thepicture 4. In addition, the pictures 9, 10, and 11 which are B-pictureseach refer to the picture 8 and the picture 12. It is to be noted thatthe pictures 4 and 12 which are I-pictures do not refer to otherpictures.

A picture to be referred to needs to be encoded prior to a picture whichrefers in video encoding, and a picture to be referred to needs to bedecoded prior to a picture which refers in video decoding. Thus, whenthe prediction structure (frame reference relation) illustrated in (c)in FIG. 1 is used for the inputting order illustrated in (a) in FIG. 1,encoding or decoding is performed in encoding (decoding) orderillustrated in (b) in FIG. 1. More specifically, after the picture 4which is the RAP picture is encoded, the picture 8 is encoded withreferenced to the picture 4, and the pictures 5, 6, and 7 are encoded inorder with referenced to the pictures 4 and 8. Subsequently, the picture12 which is the RAP picture is encoded, and the pictures 9, 10, and 11are encoded in order with referenced to the pictures 8 and 12. Much thesame is true on decoding. After the picture 4 which is the RAP pictureis decoded, the picture 8 is decoded with referenced to the picture 4,and the pictures 5, 6, and 7 are decoded in order with referenced to thepictures 4 and 8. Subsequently, the picture 12 which is the RAP pictureis decoded, and the pictures 9, 10, and 11 are decoded in order withreferenced to the pictures 8 and 12.

The following describes a leading picture which is introduced in highefficiency video coding (hereinafter referred to as an HEVC scheme)which is a video coding method that is under way of standardization. Aleading picture is a picture which is subsequent to the RAP picture indecoding order and precedes the RAP picture in outputting order (displayorder). In the examples illustrated in (a) and (b) in FIG. 1, thepictures 9, 10, and 11 are subsequent, in decoding order, to the picture12 which is the RAP picture. In addition, the pictures 9, 10, and 11precede, in outputting order, the picture 12 which is the RAP picture.In other words, the pictures 9, 10, and 11 are leading pictures (LP)with respect to the picture 12 which is the RAP picture.

In the HEVC scheme, three types of pictures, an instantaneous decodingrefresh picture (IDR picture), a broken link access picture (BLApicture), and a clean random access picture (CRA picture), are defined,for the RAP picture.

FIG. 2 is a diagram for explaining the leading picture when the RAPpicture is the IDR picture. According to this case, the outputting orderof decoded pictures or inputting order of pictures to be encoded is thesame as the order illustrated in (a) in FIG. 1. It is to be noted that,in FIG. 2, a rectangle frame indicates a picture, a numerical characterin the frame indicates a picture number, and an alphabet indicates apicture type, in the same manner as FIG. 1.

In FIG. 2, (a) indicates a decoding order of encoded pictures, as with(b) in FIG. 1. However, (a) in FIG. 2 is different from (b) in FIG. 1 inthat the picture 12 is the IDR picture.

In FIG. 2, (b) illustrates a frame reference relation of a leadingpicture with respect to the IDR picture, which is used in the interframe prediction. A frame reference relation of the pictures 9, 10, and11 which are the leading pictures with respect to the picture 12 whichis the IDR picture is illustrated as an example. The pictures 9, 10, and11 which are the leading pictures each refer to only the picture 12. Itis forbidden for a picture that is subsequent to an IDR picture indecoding order to refer to a picture which is previous to (preceding)the IDR picture in decoding order. For example, the pictures 9, 10, and11 cannot refer to the picture 8 or the like which is previous to thepicture 12 in decoding order.

In other words, a leading picture of an IDR picture cannot refer to apicture previous to the IDR picture in decoding order. It is to be notedthat, in the case of the RAP picture other than the IDR picture; thatis, the CRA picture and the BLA picture, it is not forbidden for theleading picture with respect to the RAP picture to refer to a picturewhich is previous to the RAP picture in decoding order.

The following describes the BLA picture. The BLA picture is a picturewhich is generated when recorded content is edited, for example. Morespecifically, when video is edited such that recorded content 1 isdivided into A, B, and C, B is deleted, and A and C are combined, theRAP picture at the top of C becomes the BLA picture. It is to be notedthat when the content 1 is divided into A, B, and C, the content 1 isdivided at the position of the RAP picture. In addition, as anotherexample, when record content 1 and record content 2 are combined, theRAP picture at the top of the recorded content 2 can be the BLA picture.

FIG. 3 is a diagram for explaining the leading picture when the RAPpicture is the BLA picture. According to this case, the outputting orderof decoded pictures or the inputting order of pictures to be encoded isthe same as the order illustrated in (a) in FIG. 1. It is to be notedthat, also in FIG. 3, a rectangle frame indicates a picture, a numericalcharacter in the frame indicates a picture number, and an alphabetindicates a picture type, in the same manner as FIG. 1.

In FIG. 3, (a) indicates a decoding order of encoded pictures, as with(b) in FIG. 1. However, (a) in FIG. 3 is different from (b) in FIG. 1 inthat the picture 12 is the BLA picture. In addition, as described above,the bit stream from the picture 1′ to the picture 8′ and the bit streamfrom the picture 9 to the picture 16 lack continuity, correspond, forexample, to A in the content 1 and C in the content 1, and simplycombined by editing. In this point, too, (a) in FIG. 3 is different from(b) in FIG. 1.

In FIG. 3, (b) illustrates a frame reference relation of the leadingpictures with respect to the BLA picture, which is used in the interframe prediction. A frame reference relation of the pictures 9, 10, and11 which are the leading pictures with respect to the picture 12 whichis the BLA picture is illustrated as an example. As illustrated in (b)in FIG. 3, the leading pictures 9, 10, and 11 with respect to thepicture 12 refer to the picture 8′ and the picture 12. At this time,decoding is formally possible using the picture 8′ and the picture 12;however, the picture 8′ is a picture having no relation, for example, tothe pictures 9, 10, and 11 which are combined by editing. In otherwords, since the picture 8′ is not a proper reference picture for thepictures 9, 10, and 11, it is not possible to obtain a proper decodedimage. It can be said that the reference structure of the leadingpictures 9, 10, and 11 is broken (broken link).

It is to be noted that the CRA picture is a RAP picture other than theIDR picture and the BLA picture. In the case of the CRA picture, sincethe leading picture with respect to the CRA picture can refer to apicture which is previous to the CRA picture in decoding order, thecoding efficiency is greater than the case of the IDR picture. Inaddition, since the reference structure is not broken unlike the leadingpicture of the BLA picture, it is possible to decode all the leadingpictures at the time of normal reproduction.

Classification of the RAP picture has been described above, and thefollowing describes classification of the leading picture.

Two types of leading pictures are defined; that is, a random accessdecodable leading (RADL) picture and a random access skipped leading(RASL) picture. When jumping into (random accessing) a RAP picture anddecoding from the RAP picture (at the time of random accessreproduction), the RADL picture is a decodable leading picture. On theother hand, the RASL picture is a leading picture which cannot bedecoded at the time of random access reproduction.

FIG. 4 is a diagram for explaining classification of the leadingpicture. According to this case, the outputting order of decodedpictures or the inputting order of pictures to be encoded is the same asthe order illustrated in (a) in FIG. 1. It is to be noted that, in FIG.4, a rectangle frame indicates a picture, a numerical character in theframe indicates a picture number, and an alphabet indicates a picturetype, in the same manner as FIG. 1. In FIG. 4, (a) indicates a decodingorder of encoded pictures, as with (b) in FIG. 1. However, (a) in FIG. 4is different from (b) in FIG. 1 in that the picture 12 is a CRA picture,and the leading pictures 9, 10, and 11 are subdivided into a RASLpicture and RADL pictures.

In FIG. 4, (b) illustrates an example of a reference picture of theleading pictures 9, 10, and 11 with respect to the CRA picture. Thepicture 9 refers to the picture 8 and the picture 12. As a result, thepicture 8 cannot be used as a reference picture when decoding is startedfrom the picture 12 (at the time of random accessing to the picture 12),and thus the picture 9 cannot be decoded. Thus, the picture 9 becomes aRASL picture. On the other hand, the pictures 10 and 11 refer only tothe picture 12. As a result, the picture 12 can be used as a referencepicture when decoding is started from the picture 12, and thus it ispossible to decode the pictures 10 and 11. Thus, the pictures 10 and 11become RADL pictures.

As described above, the leading picture of the IDR picture cannot referto a picture previous to the IDR picture in decoding order. Thus, theleading picture of the IDR picture always becomes the RADL picture.Alternatively, there is no leading picture with respect to the IDRpicture in some cases. The leading picture of the CRA picture and theBLA picture includes only the RADL picture, only the RASL picture, or amix of the RASL picture and the RADL picture or neither of them in somecases.

In other words, which type (RADL picture or RASL picture) of leadingpicture is present differs according to the type of RAP picture.Furthermore, there is the case where there is no leading picture. When aRAP picture is decoded, it is beneficial for the decoding processing ifinformation of a type of leading picture with respect to the RAPpicture, or information indicating that there is no leading picture withrespect to the RAP picture, is obtained.

In view of the above, a NAL unit type is described in a NAL unit header(the above-described header information) of a network abstraction layer(NAL) unit. The NAL unit is a unit for sectioning encoded data, andincludes two-byte (fixed length) NAL unit header and payload. The NALunit header includes a NAL unit type which is an identifier indicating atype of the NAL unit. The payload includes raw byte sequence payload(RBSP) which is encoded data. In order to perform byte alignment on theRBSP (multiple of 8 bits), trailing bit of 1 to 8 bits is added at theend of RBSP.

By referring to the NAL unit type, it is possible to identify the typeof data included in the payload of the NAL unit. In addition, the NALunit can include not only video data but also an encoded parameter suchas a sequence parameter set (SPS), a picture parameter set (PPS), and soon. When a NAL unit includes video data, the NAL unit is configured toinclude, in the payload, video data in a unit called a slice resultingfrom dividing a picture. It is to be noted that a NAL unit type of a NALunit of slices which belong to the same picture has the same value.

FIG. 5 is a diagram which illustrates a NAL unit type related to the RAPpicture and the leading picture.

For the NAL unit type of a NAL unit including the RAP picture (slice ofRAP picture), 16 to 18 are used when the RAP picture is a BLA picture,19 and 20 are used when the RAP picture is an IDR picture, and 21 isused when the RAP picture is a CRA picture.

More specifically, when BLA_W_LP (16) is set to a NAL unit type, the NALunit type indicates that there is a possibility that the leading pictureof the BLA picture is included in an image signal. When BLA_W_RADL (17)is set to a NAL unit type, the NAL unit type indicates that there is apossibility that the leading picture of the BLA picture is included inan image signal, and the leading picture is a decodable RADL pictureonly (not a RASL picture). When BLA_N_RADL (18) is set to a NAL unittype, the NAL unit type indicates that there is a possibility that theleading picture of the BLA picture is not included in an image signal.

In addition, when IDR_W_RADL (19) is set to a NAL unit type, the NALunit type indicates that there is a possibility that the leading pictureof the IDR picture is included in an image signal, and the leadingpicture is a decodable RADL only (not a RASL picture). When IDR_N_LP(20) is set to a NAL unit type, the NAL unit type indicates that thereis a possibility that the leading picture of the IDR picture is notincluded in an image signal.

In addition, when CRA_NUT (21) is set to a NAL unit type, the NAL unittype indicates that there is a possibility that the leading picture ofthe CRA picture is included in an image signal, and the leading picturecan either be a decodable RADL or a RASL that cannot be decoded.

In addition, as described above, two types of leading pictures aredefined; that is, a RADL picture and a RASL picture. In the NAL unittype of a NAL unit which includes a RADL picture (slice of a RADLpicture), 6 (or 7) that means a RADL picture is inserted. In addition,in the NAL unit type of a NAL unit which includes a RASL picture (sliceof a RASL picture), 8 (or 9) that means a RASL picture is inserted.

However, a parameter which indicates the restriction on the referencerelation such as the NAL unit types described above might be incorrectin some cases. In such a case, inconsistency arises in the referencerelation in image signals, and there is a problem that an error islikely to occur due to the inconsistency in the reference relation withthe image processing methods according to NPL 1 and NPL 2.

In order to solve such a problem, an image processing method accordingto an aspect of the present disclosure is an image processing method forprocessing an image signal generated by encoding a plurality ofpictures, the image processing method including: obtaining a parameterfrom the image signal, the parameter indicating restriction on areference relation between at least one of the plurality of pictures andan other one of the plurality of pictures in the encoding; andperforming restriction alleviation processing for alleviating therestriction on the reference relation indicated by the parameter.

With this, even when restriction on a reference relation indicated by aparameter, such as a NAL unit type and a temporal nesting flag, isfalsely strict because the parameter is incorrect, the restriction onthe reference relation is alleviated. It is therefore possible tosuppress an error resulting from inconsistency that occurs between therestriction on the reference relation indicated by the parameter and anactual reference relation in image signals (relation indicated by anumber of a reference picture, or the like) or a reference relationindicated by another parameter (different NAL unit type). Thus, it ispossible to prevent decoding on an image signal from ending due to anerror, for example.

In addition, the image signal may include a plurality of networkabstraction layer (NAL) units which are signal units, in the obtaining,a NAL unit type included in a target NAL unit to be processed which isany one of the plurality of NAL units may be obtained as the parameter,and in the performing, when the NAL unit type is a first NAL unit type,a second NAL unit type may be applied to the target NAL unit instead ofthe first NAL unit type, the second NAL unit type defining alleviatedrestriction which is less strict than restriction on a referencerelation defined by the first NAL unit type.

With this, even when the first NAL unit type included in an image signalis incorrect, the second NAL unit type in which restriction on areference relation is alleviated is applied to a target NAL unit whichis a target for processing, and thus it is possible to properly suppressthe above-described inconsistency from occurring.

In addition, in the performing, when the first NAL unit type definesforbidding a reference relation in which a subsequent picture subsequentin decoding order to a broken link access (BLA) picture which is astarting point of random access reproduction and corresponds to thetarget NAL unit refers, as a leading picture which precedes the BLApicture in display order, to an other picture, the second NAL unit typewhich defines permitting the reference relation may be applied to thetarget NAL unit, instead of the first NAL unit type.

With this, instead of the first NAL unit type which does not permit thepresence of a leading picture (for example, BLA_N_LP), the second NALunit type which permits the presence of a leading picture (for example,BLA_W_RADL or BLA_W_LP) is applied to the target NAL unit. Accordingly,even when a NAL unit type of a subsequent NAL unit that is subsequent tothe target NAL unit in decoding order indicates a leading picture, it ispossible to suppress occurrence of inconsistency of the NAL unit typebetween the target NAL unit and the subsequent NAL unit.

In addition, in the performing, when the first NAL unit type definesforbidding a reference relation in which the subsequent picture refers,as the leading picture, to a picture which precedes the BLA picture indecoding order, the second NAL unit type which defines permitting thereference relation may be applied to the target NAL unit, instead of thefirst NAL unit type.

With this, instead of the first NAL unit type which does not permit thepresence of a RASL picture which is a leading picture (for example,BLA_N_LP), the second NAL unit type which permits the presence of theRASL picture (for example, BLA_W_RADL or BLA_W_LP) is applied to thetarget NAL unit. Accordingly, even when a NAL unit type of a subsequentNAL unit that is subsequent to the target NAL unit in decoding orderindicates a RASL picture, it is possible to suppress occurrence ofinconsistency of the NAL unit type between the target NAL unit and thesubsequent NAL unit.

In addition, in the performing, when the first NAL unit type definesforbidding a reference relation in which a subsequent picture subsequentin decoding order to an instantaneous decoding refresh (IDR) picturewhich is a starting point of random access reproduction and correspondsto the target NAL unit refers, as a leading picture which precedes theIDR picture in display order, to an other picture, the second NAL unittype which defines permitting the reference relation may be applied tothe target NAL unit, instead of the first NAL unit type.

With this, instead of the first NAL unit type which does not permit thepresence of a leading picture (for example, IDR_N_LP), the second NALunit type which permits the presence of a leading picture (for example,IDR_W_RADL) is applied to the target NAL unit. Accordingly, even when aNAL unit type of a subsequent NAL unit that is subsequent to the targetNAL unit in decoding order indicates a leading picture, it is possibleto suppress occurrence of inconsistency of the NAL unit type between thetarget NAL unit and the subsequent NAL unit.

In addition, in the performing, when the first NAL unit type definesthat a picture corresponding to the target NAL unit is: a random accessskipped leading (RASL) picture that (i) is a leading picture which issubsequent in decoding order to a random access point (RAP) picture thatis a starting point of the random access reproduction and which precedesthe RAP picture in display order and (ii) has a reference relation inwhich decoding is impossible when the random access reproduction isperformed, the second NAL unit type may be applied to the target NALunit instead of the first NAL unit type, and the picture is decoded, thesecond NAL unit type defining that the picture is a random accessdecodable leading (RADL) picture that is the leading picture and has areference relation in which decoding is possible even when the randomaccess reproduction is performed.

With this, instead of the first NAL unit type (for example, RASL), thesecond NAL unit type (for example, RADL) is applied to the target NALunit. Accordingly, even when a picture of a target NAL unit that issubsequent to the RAP picture in decoding order is set, falsely as aRASL picture, so as not to be decoded at the time of random accessreproduction in spite of the fact that the picture does not refer toother pictures which precede the RAP picture in decoding order, it ispossible to properly decode the picture of the target NAL unit at thetime of random access reproduction.

In addition, when the picture is decoded, reference picture informationcorresponding to the picture may be decoded, when a reference pictureidentified by the decoded reference picture information is stored in abuffer, the picture may be decoded by referring to the referencepicture, and when the reference picture is not stored in the buffer, analternative picture may be generated, and the picture may be decoded byreferring to the alternative picture as the reference picture.

For example, since the second NAL unit type, instead of the first NALunit type, is applied to a target NAL unit, the second NAL unit type(for example, RADL) which is false is applied to the target NAL unitwhen the first NAL unit type (for example, RASL) is correct.Accordingly, when a picture of a target NAL unit, which is actually aRASL picture, is reproduced as a RADL picture in the random accessreproduction, a reference picture of the picture is not stored in abuffer in some cases. In view of the above, when the reference pictureis not stored in the buffer as described above, an alternative pictureis generated and referred to as the reference picture, and thus it ispossible to properly decode the picture of the target NAL unit.

In addition, the plurality of pictures may be each classified into anyone of a plurality of layers, and encoded without referring to a picturewhich belongs to a layer of a higher level than a layer to which thepicture belongs, in the obtaining, a flag may be obtained as theparameter, the flag forbidding a target picture to be processed amongthe plurality of pictures from referring to, in a predetermined state, alower level picture which belongs to a layer of a lower level than alayer to which the target picture belongs, and in the performing,whether or not the lower level picture is stored in a buffer may bedetermined, and when it is determined that the lower level picture isnot stored in the buffer, the lower level picture may be decoded, andthe target picture may be decoded by referring to the decoded lowerlevel picture without complying with the flag.

For example, in the case where the above-described flag (for example, atemporal nesting flag which indicates true (1)) is falsely included inan image signal in spite of the fact that a target picture to beprocessed refers to a lower level picture, the target picture cannotrefer to the lower level picture because the lower level picture is notstored in the buffer, in some cases. In view of the above, when a lowerlevel picture is not stored in the buffer, since the lower level pictureis decoded and alleviation allows referring to the lower level picturewhich is forbidden by the flag as described above, it is possible toproperly decode the target picture.

In addition, the image signal may include a plurality of NAL units whichare signal units, and each of pictures corresponding to a different oneof the plurality of NAL units may be classified into any one of aplurality of layers, and encoded without referring to a picture whichbelongs to a layer of a higher level than a layer to which the picturebelongs, in the obtaining, a NAL unit type may be obtained as theparameter, the NAL unit type being included in a NAL unit correspondingto a target picture to be processed among the plurality of pictures, anddefining forbidding a reference relation in which the target picturerefers to a specific picture which belongs to a same layer as the targetpicture, the image processing method further includes, determining, whenrestriction is imposed on a layer to which a picture to be decoded amongthe plurality of pictures belongs, whether or not upswitching foralleviating the restriction on the layer is possible for the targetpicture, based on the obtained NAL unit type and the layer to which thetarget picture belongs, and in the performing, when it is determinedthat the upswitching is possible, whether or not the specific picture isstored in a buffer may be determined, the specific picture being areference picture to which the target picture refers, and when it isdetermined that the specific picture is not stored in the buffer, thereference picture may be decoded, and the target picture may be decodedby referring to the reference picture, which is forbidden by the NALunit type.

For example, when a NAL unit type, such as TSA or STSA, which forbids areference relation in which a specific picture is referred to is falselyincluded in a NAL unit, in spite of the fact that a target picturerefers to the specific picture, there is the case where the targetpicture to be processed cannot refer to the specific picture at the timeof upswitching because the specific picture is not stored in a buffer.In view of the above, when a specific picture is not stored in thebuffer as a reference picture, since the reference picture is decodedand alleviation allows referring to the reference picture which isforbidden by the NAL unit type as described above, it is possible toproperly decode the target picture.

In addition, in the determining, when: a temporal sub-layer access (TSA)picture is obtained as the NAL unit type, the TSA defining adding, as alayer to which a picture to be decoded among the plurality of picturesbelongs, a layer to which the target picture belongs and a layer of ahigher level than the layer to which the target picture belongs; and thelayer to which the target picture belongs is of a highest level amonglayers to which pictures decoded prior to the target picture in decodingorder belong, it may be determined that the upswitching is possible.

With this, it is possible to properly perform upswitching according toTSA.

In addition, in the determining, when: a step-wise temporal sub-layeraccess (STSA) picture is obtained as the NAL unit type, the STSAdefining adding, as a layer to which a picture to be decoded among theplurality of pictures belongs, only a layer to which the target picturebelongs; and the layer to which the target picture belongs is of ahighest level among layers to which pictures decoded prior to the targetpicture in decoding order belong, it may be determined that theupswitching is possible.

With this, it is possible to properly perform upswitching according toSTSA.

In addition, in the performing, a NAL unit including the second NAL unittype may be generated by replacing the first NAL unit type included inthe target NAL unit with the second NAL unit type.

With this, since an image signal is edited by a new image signal with analleviated reference relation, an image decoding apparatus, for example,can perform decoding of an image in which occurrence of an error issuppressed, without performing special processing.

These general and specific aspects may be implemented using a system, amethod, an integrated circuit, a computer program, or acomputer-readable recording medium such as a CD-ROM, or any combinationof systems, methods, integrated circuits, computer programs, orcomputer-readable recording media.

Hereinafter, embodiments are specifically described with reference tothe Drawings.

Each of the exemplary embodiments described below shows a general orspecific example. The numerical values, shapes, materials, structuralelements, the arrangement and connection of the structural elements,steps, the processing order of the steps etc. shown in the followingembodiments are mere examples, and therefore do not limit the scope ofthe Claims. Therefore, among the structural elements in the followingembodiments, structural elements not recited in any one of theindependent claims are described as arbitrary structural elements.

Embodiment 1

An embodiment according to the present disclosure will be describedbelow with reference to the drawings.

First, an overall operation of a decoder (image processing apparatus)which decodes a bit stream encoded by HEVC scheme, which is assumed bythe inventors, will be described with reference to FIG. 6.

FIG. 6 is a flowchart which illustrates an operation of a decoder whichis an image processing apparatus according to this exemplary embodiment.

The decoder performs syntax analysis of a NAL unit included in an imagesignal, and obtains a NAL unit type from the NAL unit (Step S901). Next,the decoder determines whether or not the NAL unit type (NALUT) is a RAP(Step S902). More specifically, the decoder determines whether or notthe NAL unit type is any one of 16 to 21. Here, when it is determinedthat that the NAL unit type is any one 16 to 21, the decoder recognizesthat a picture (slice) included in the NAL unit is the RAP picture (Yesin Step S902). Then, the decoder records the RAP type (NAL unit type) onan internal memory, and decodes the RAP picture (Step S903). Then, whena subsequent picture that is subsequent to the RAP picture in decodingorder is present, the decoder performs the processes from Step S901again and tries to decode the subsequent picture.

When it is determined that that the NAL unit type is not the RAP (No inStep S902), the decoder determines which type a NAL unit type (NALUT)obtained from the NAL unit related to the subsequent picture that issubsequent to the RAP picture is (Step S904).

Here, when it is determined that the NAL unit type is 6 or 7, in otherwords, a RADL (RADL in Step S904), the decoder determines whether or notthe RAP type (NAL unit type) recorded in Step S903 permits the presenceof a RADL picture (Step S905). When the presence of the RADL picture ispermitted (Yes in Step S905), the decoder decodes the subsequent picture(leading picture) which is the RADL picture (Step S907). When thepresence of the RADL picture is not permitted (No in Step S905), thedecoder performs predetermined error processing because the presence ofthe RADL picture means an error (Step S908).

When the NAL unit type is 8 or 9 (RASL), in other words, a RASL in StepS904, the decoder determines whether or not the RAP type (NAL unit type)recorded in Step S903 permits the presence of the RASL picture (StepS906). When the presence of the RASL picture is permitted (Yes in StepS906), the decoder deletes the subsequent picture (leading picture)which is the RASL picture without decoding it (Step S909). It is to benoted that, the case where the presence of the RASL picture is permittedis the case where the RAP type (NAL unit type) is a type in which a RASLpicture can be present, such as CRA and BLA. On the other hand, when thepresence of the RASL picture is not permitted (No in Step S906), thedecoder performs a predetermined error processing because the presenceof the RASL picture means an error (Step S910). It is to be noted that,the case where the presence of the RASL picture is not permitted is thecase where the RAP type (NAL unit type) is a type in which the RASLpicture is not permitted to be present, such as IDR.

FIG. 7 is a block diagram which illustrates a configuration of an imageprocessing apparatus which is the decoder according to this exemplaryembodiment.

A decoder (image decoding apparatus) 100 includes: a coded picturebuffer (CPB) 102; a NAL unit syntax analysis unit 103; a data decodingunit 104; and a decoded picture buffer (DPB) 105, as illustrated in FIG.7.

The CPB buffer 102 is a buffer which holds a bit stream 101 that is animage signal provided to the decoder 100. The CPB buffer 102 outputs abit stream to the NAL unit syntax analysis unit 103 with a predeterminedtiming for each picture (access unit).

The NAL unit syntax analysis unit 103 analyzes syntax of a NAL unitincluded in the bit stream. More specifically, the NAL unit syntaxanalysis unit 103 analyzes a NAL unit header to obtain information suchas a NAL unit type and so on. Then, the NAL unit syntax analysis unit103 transmits a payload included in the NAL unit to the data decodingunit 104 or the like.

The data decoding unit 104 decodes the payload (encoded video data)transmitted from the NAL unit syntax analysis unit 103.

The DPB buffer 105 holds the picture (uncompressed image 106) decoded bythe data decoding unit 104. The decoded picture is inputted to the DPBbuffer in decoding order indicated in (b) in FIG. 1, for example, andoutputted to a display device or the like in outputting order in (a) inFIG. 1.

FIG. 8 is a flowchart which illustrates an operation of the NAL unitsyntax analysis unit 103 of the decoder 100 according to this exemplaryembodiment. Upon receiving an input of the picture (access unit) fromthe CPB buffer 102, the NAL unit syntax analysis unit 103 performssyntax analysis sequentially on each NAL unit included in the accessunit Then, the NAL unit syntax analysis unit 103 obtains a NAL unit typefrom the NAL unit in the same manner as in Step S901 illustrated in FIG.6 (Step S101). It is to be noted that a NAL unit type of a NAL unit ofslices which belong to the same picture has the same value. The NAL unitsyntax analysis unit 103, at this time, sets the obtained NAL unit typeas a first NAL unit type. Next, the NAL unit syntax analysis unit 103determines whether or not the obtained NAL unit type is a RAP, in thesame manner as in Step S902 illustrated in FIG. 6 (Step S102). When theobtained NAL unit type is the RAP (Yes in Step S102), the NAL unitsyntax analysis unit 103 determines which one of BLA, IDR, and CAR theNAL unit type is (Step S121).

When the NAL unit type is any one of three types of BLA indicated byvalues of 16 to 18 described above (BLA in Step S121), the NAL unitsyntax analysis unit 103 sets BLA_W_LP (16) as a second NAL unit type(Step S122). BLA_W_LP is a NAL unit type which permits the presence of aleading picture such as a RASL picture and a RADL picture, for the BLApicture as described above. It is to be noted that the NAL unit syntaxanalysis unit 103 may check the NAL unit type of a subsequent leadingpicture and set BLA_W_RADL as the second NAL unit type when the RASLpicture is not present.

When the NAL unit type is one of two types of IDR indicated by 19 or 20as described above (IDR in Step S121), the NAL unit syntax analysis unit103 sets IDR_W_RADL (19) as the second NAL unit type. IDR_W_RADL is aNAL unit type which permits the presence of a leading picture (onlyRADL) for the IDR picture as described above.

When the NAL unit type is CRA indicated by a value of 21 as describedabove (CRA in Step S121), the NAL unit syntax analysis unit 103 sets CRAas the second NAL unit type (Step S124).

It is to be noted that the processes in each step indicated by theflowchart in FIG. 8 are executed in Step S901 illustrated in FIG. 6.More specifically, for the NAL unit of the RAP analyzed in Step S901illustrated in FIG. 6, the second NAL unit type is applied instead ofthe NAL unit type included in the NAL unit (the first NAL unit type).Accordingly, in Step S903 illustrated in FIG. 6, the second NAL unittype is recorded as the RAP type.

As described above, the decoder 100 sets, as the second NAL unit type,the NAL unit type which permits the leading picture and performs adecoding operation based on the second NAL unit type. By the editing ofthe NAL unit type into a NAL unit type which permits a leading picture,even when an error stream in which a leading picture is present in spiteof the fact that the leading picture is not permitted, the decoder 100is capable of decoding the error stream. In other words, it is possibleto suppress execution of the processes of Step S908 and Step S910illustrated in FIG. 6.

In sum, the image processing method according to this exemplaryembodiment includes: obtaining a NAL unit type indicating restriction ona reference relation from an image signal; and executing restrictionalleviation processing for alleviating the restriction on the referencerelation indicated by the NAL unit type.

With this, when the restriction on a reference relation indicated by theNAL unit type is falsely strict because the NAL unit type is incorrect,the restriction on the reference relation is alleviated. It is thereforepossible to suppress an error resulting from inconsistency that occursbetween the restriction on the reference relation indicated by the NALunit type of a NAL unit and the reference relation indicated by the NALunit type of another NAL unit. Thus, it is possible to prevent decodingon an image signal from ending due to an error.

For example, in obtaining of the NAL unit type, a NAL unit type includedin a target NAL unit which is a target for processing among a pluralityof NAL units is obtained. Then, in executing of the restrictionalleviation processing, when the NAL unit type is the first NAL unittype, the second NAL unit type in which alleviated restriction which isless strict than restriction on the reference relation defined for thefirst NAL unit type is defined is applied to the target NAL unit,instead of the first NAL unit type.

With this, even when the first NAL unit type included in an image signalis incorrect, the second NAL unit type in which the restriction on areference relation is alleviated is applied to the target NAL unit, andthus it is possible to properly suppress the above-describedinconsistency from occurring.

More specifically, the first NAL unit type forbids the referencerelation in which a subsequent picture that is subsequent to, indecoding order, a BLA picture which is a starting point of random accessreproduction and corresponds to a target NAL unit, as a leading picturewhich precedes the BLA picture in display order, refers to otherpictures. In this case, in executing of the restriction alleviationprocessing, the second NAL unit type which defines permitting thereference relation is applied to the target NAL unit, instead of thefirst NAL unit type.

With this, instead of the first NAL unit type (BLA_N_LP) which does notpermit the presence of a leading picture, the second NAL unit type(BLA_W_RADL or BLA_W_LP) which permits the presence of a leading pictureis applied to the target NAL unit. Accordingly, even when a NAL unittype of a subsequent NAL unit that is subsequent to the target NAL unitin decoding order indicates a leading picture, it is possible tosuppress occurrence of inconsistency of the NAL unit type between thetarget NAL unit and the subsequent NAL unit.

In addition, the first NAL unit type forbids a reference relation inwhich a subsequent picture refers, as a leading picture, to a picturepreceding the BLA picture in decoding order. Also in this case, inexecuting of the restriction alleviation processing, the second NAL unittype which defines permitting the reference relation is applied to thetarget NAL unit, instead of the first NAL unit type.

With this, instead of the first NAL unit type (BLA_N_LP or BLA_W_RADL)which does not permit the presence of a RASL picture which is a leadingpicture that refers to a picture preceding the BLA picture in decodingorder, the second NAL unit type (BLA_W_LP) which permits the presence ofthe RASL picture is applied to the target NAL unit. Accordingly, evenwhen a NAL unit type of a subsequent NAL unit that is subsequent to thetarget NAL unit in decoding order indicates a RASL picture, it ispossible to suppress occurrence of inconsistency of the NAL unit typebetween the target NAL unit and the subsequent NAL unit.

In addition, the first NAL unit type forbids a reference relation inwhich a subsequent picture that is subsequent, in decoding order, to anIDR picture which is a starting point of random access reproduction andcorresponds to a target NAL unit refers, as a leading picture whichprecedes the IDR picture in decoding order, to other pictures. Also inthis case, in executing of the restriction alleviation processing, thesecond NAL unit type which defines permitting the reference relation isapplied to the target NAL unit, instead of the first NAL unit type.

With this, instead of the first NAL unit type (IDR_N_LP) which does notpermit the presence of a leading picture, the second NAL unit type(IDR_W_RADL) which permits the presence of a leading picture is appliedto the target NAL unit. Accordingly, even when a NAL unit type of asubsequent NAL unit that is subsequent in decoding order to the targetNAL unit indicates a leading picture, it is possible to suppressoccurrence of inconsistency of the NAL unit type between the target NALunit and the subsequent NAL unit.

Embodiment 2

In Embodiment 1 described above, the second NAL unit type is applied tothe NAL unit of the RAP, instead of the NAL unit type included in theNAL unit (the first NAL unit type). In Embodiment 2, the second NAL unittype is applied to a NAL unit other than the RAP, instead of the NALunit type included in the NAL unit (the first NAL unit type).

FIG. 9 is a flowchart which illustrates an operation of the NAL unitsyntax analysis unit of the decoder 100 according to this exemplaryembodiment.

The NAL unit syntax analysis unit 103 obtains a NAL unit type from a NALunit in the same manner as in Step S902 illustrated in FIG. 6 (StepS201). The NAL unit syntax analysis unit 103, at this time, sets theobtained NAL unit type as a first NAL unit type. In addition, the NALunit syntax analysis unit 103 determines whether or not the obtained NALunit type is a RAP, in the same manner as in Step S902 illustrated inFIG. 6 (Step S202). When the obtained NAL unit type is not a RAP (No inStep S202), the NAL unit syntax analysis unit 103 determines whether ornot the NAL unit type is a RASL (Step S225). When the obtained NAL unittype is the RASL (Yes in Step S225), the NAL unit syntax analysis unit103 sets the RADL as the second NAL unit type (Step S226) Morespecifically, the RASL which is the first NAL unit type is changed tothe RADL which is the second NAL unit type.

In a bit stream which includes an error (error stream), there is apossibility that a NAL unit type indicating that a picture is a RASLpicture is assigned, in spite of the fact that the picture is a picturethat can be decoded. For that reason, the processes of Step S226 isperformed. Although the second NAL unit type is set as described above,there is a possibility that decoding is really impossible, and thus theNAL unit syntax analysis unit 103 decodes video (Step S227). Morespecifically, the NAL unit syntax analysis unit 103 decodes a sliceheader included in the NAL unit, and obtains a number of a referencepicture (reference picture information) which is indicated in the sliceheader. Then, the NAL unit syntax analysis unit 103 determines whetheran error occurs or not, based on the number of the reference picture(Step S228). More specifically, the NAL unit syntax analysis unit 103determines whether or not a picture corresponding to the number of thereference picture (picture number) is present in the DPB buffer 105, anddetermines that an error will occur when the reference picture is notpresent. When it is determined that an error will occur (Yes in StepS228), the NAL unit syntax analysis unit 103 returns the second NAL unittype to the first NAL unit type (Step S229). More specifically, the RADLwhich is the second NAL unit type is returned to the RASL which is thefirst NAL unit type.

It is to be noted that the process in each step indicated by theflowchart in FIG. 9 is executed in Step S901 illustrated in FIG. 6. Morespecifically, for a NAL unit which is analyzed in Step S901 illustratedin FIG. 6 and is not the RAP, the second NAL unit type=RADL is appliedinstead of the NAL unit type (first NAL unit type=RASL) included in theNAL unit, unless it is determined that an error will occur in Step S228illustrated in FIG. 9.

In other words, according to this exemplary embodiment, the first NALunit type indicates that a picture corresponding to the target NAL unitis RASL picture which is: a leading picture that (i) is subsequent, indecoding order, to the RAP picture which is the starting point of therandom access reproduction and (ii) precedes the RAP picture in displayorder; and has a reference relation in which decoding is impossible whenthe random access reproduction is performed. In this case, in executingof the restriction alleviation processing, the second NAL unit typewhich defines that the picture is a RADL picture which is a leadingpicture and has a reference relation in which decoding is possible evenwhen the random access reproduction is performed is applied, instead ofthe first NAL unit type, to the target NAL unit and the picture isdecoded.

With this, instead of the first NAL unit type (RASL), the second NALunit type (RADL) is applied to the target NAL unit. Accordingly, evenwhen a picture of a target NAL unit which is subsequent to the RAPpicture in decoding order is set, falsely as a RASL picture, so as notto be decoded at the time of random access reproduction in spite of thefact that the picture does not refer to other pictures which precede theRAP picture in decoding order, it is possible to properly decode thepicture of the target NAL unit at the time of random accessreproduction.

According to the above-described example, the NAL unit type is returnedto the RASL which is the type before change when it is determined thatan error will occur in Step S228, however, the NAL unit type is notnecessarily be returned. In this case, the data decoding unit 104 of thedecoder 100 performs video decoding with error correction. Morespecifically, the data decoding unit 104 generates an alternativepicture as a reference picture, and decodes the RADL picture withreference to the reference picture. The alternative picture is, forexample, a gray image having 0 chromaticity and luminance that is amedian value of possible values.

More specifically, in decoding of a RADL picture, reference pictureinformation corresponding to the RADL picture is decoded, and when areference picture which is identified by the decoded reference pictureinformation is stored in the DPB buffer 105, the RADL picture is decodedby referring to the reference picture. On the other hand, when thereference picture is not stored in the DPB buffer 105, an alternativepicture is generated, and the RADL picture is decoded by referring tothe alternative picture as the reference picture.

For example, in Step S226, since the second NAL unit type is applied tothe target NAL unit instead of the first NAL unit type, when the firstNAL unit type (RASL) is correct, the second NAL unit type (RADL) whichis incorrect is applied to the target NAL unit. Accordingly, when apicture of the target NAL unit, which is actually a RASL picture, isreproduced as a RADL picture in the random access reproduction, areference picture of the picture is not stored in the DPB buffer 105 insome cases. In view of the above, when the reference picture is notstored in the DPB buffer 105 as described above, an alternative pictureis generated and referred to as the reference picture, and thus it ispossible to properly decode the picture of the target NAL unit.

In addition, the decoder 100, without returning the NAL unit type to theRASL that is a type before change, may output a decoded RAP picturecorresponding to the RADL picture instead of the RADL picture, as videodecoding with error correction. Alternatively, the decoder 100 mayoutput a picture which is decoded most recently instead of the RADLpicture, or may not output a picture. In this case, in drawing means,frame update is not performed for the error picture (RADL picture), andan image drawn immediately before is displayed as it is.

(Modification)

According to Embodiment 2 described above, it is determined whether theNAL unit type is RAP or RASL, however, this determination is notnecessarily been performed. In this modification, a picture of a NALunit to which an incorrect NAL unit type is applied is decoded withoutperforming the determination processes described above.

FIG. 10 is a flowchart illustrating operations performed by the imagedecoding apparatus 100 according to this Modification.

The NAL unit syntax analysis unit 103 obtains a NAL unit type from a NALunit in the same manner as in Step S901 illustrated in FIG. 6 (StepS201). Next, the data decoding unit 104 performs the above-describedvideo decoding with error correction, when the NAL unit type is set as aRADL in spite of the fact that it is actually a RASL (Step S227). Withthis, it is possible to prevent the process from branching based on theNAL unit type, making it possible to simplify the operations of thedecoder 100.

As described in each of the Embodiments and the Modification, with thedecoder 100 which is the image processing apparatus according to anaspect of the present disclosure, it is possible to properly decode evenwhen an irrelevant NAL unit type is assigned due to an error. It is tobe noted that the present disclosure may be implemented not as the imageprocessing apparatus or the image processing method which involves imagedecoding as in each of the Embodiments and the Modification describedabove but as an image processing method or an image processing apparatuswhich rewrite a NAL unit type included in a NAL unit of an inputted bitstream, and outputs, to a decoding device, a bit stream including theNAL unit type which has been rewritten as a second bit stream. In sum,the present disclosure may be implemented as an image processing methodor an image processing apparatus which performs only editing on an imagesignal without decoding the image signal. In this case, in executingrestriction alleviation processing, a NAL unit including the second NALunit type is generated by replacing the first NAL unit type included inthe target NAL unit with the second NAL unit type.

Embodiment 3

An image processing method according to this exemplary embodiment has afeature in permitting referring to a predetermined reference picturewhen it is forbidden to refer to the predetermined reference picturebecause a NAL unit type is TSA or STSA that will be described later or atemporal nesting flag that will be described later is true.

FIG. 11 is a diagram which illustrates NAL unit types used in thisexemplary embodiment.

A temporal sub-layer access picture (TSA picture) and a step-wisetemporal sub-layer access picture (STSA picture) are pictures for whicha target layer that is a target for decoding (transmitting) can beexpanded at the time of scalable decoding (or transmitting). When theNAL unit type of a NAL unit is 2 or 3 (TSA), the NAL unit includes a TSApicture (slice). In addition, when the NAL unit type of a NAL unit is 4or 5 (STSA), the NAL unit includes a STSA picture (slice).

FIG. 12 is a diagram for describing the scalable decoding.

A temporal ID (TID) is described in a NAL unit header of a NAL unit. TIDis identification information which indicates a level (layer) of areference relation, is an integer greater than or equal to 0, andindicates that the importance of the NAL unit increases as the value issmaller. For example, when there are a NAL unit M having TID=m and a NALunit N having TID=n, and m<n is satisfied, the NAL unit M does not referto the NAL unit N in inter picture prediction. The NAL unit M can bedecoded independently of the NAL unit N. A NAL unit with a small TIDvalue can be decoded independently of a NAL unit with a large TID value.

A scalable decoding is to perform decoding by thinning pictures when adecoder lacks processing capacity, when a decoder performs high-speedreproduction, or when a network system decreases a transmission rate.

In FIG. 12, each of the rectangle boxes is a picture (NAL unit), anumerical value in the box indicates a decoding order, and a TID of eachof the pictures is set to any one of three types of 0, 1, and 2. Ingeneral, when all of the pictures are decoded, the pictures are decodedin order of the 10th picture, the 11th picture, the 12th picture, the13th picture, the 14th picture, the 15th picture, the 16th picture, the17th picture, and the 18th picture.

Here, when pictures are thinned out in decoding, it is possible, forexample, to skip the pictures of TID=2 without decoding, while thepictures of TID=0 and TID=1 are decoded. The reason for this is that thepictures of TID=0 and TID=1 do not depend on the pictures of TID=2. Inthis example, the picture of TID=0 and TID=1 do not refer to the 12thpicture, the 15th picture, and the 18th picture which are pictures ofTID=2

The following describes the above-described TSA picture and the STSApicture.

It is assumed that a decoder which has received a bit stream (imagesignal) having a plurality of TIDs performs decoding by thinning outsome pictures. This decoder is assumed, for example, to decode pictureshaving TID=0 to TID=N−1, and not to decode pictures having TID=greaterthan or equal to N. Here, when the decoder has extra processing capacityor the network has extra bandwidth, the decoder decreases the extent ofthinning and resumes decoding of pictures having TID=greater than orequal to N. This switching operation is called upswitching.

The TSA picture and the STSA picture are pictures on which theupswitching can be performed.

The TSA picture is a picture that means the following. When the TID of aTSA picture is N, the TSA picture indicates that the TSA picture and apicture which is subsequent in decoding order to the TSA picture and hasa TID of greater than or equal to N (restriction-target picture) do notrefer to a picture which precedes the TSA picture in decoding order andhas a TID of greater than or equal to N (reference-forbidden picture).With this restriction, it is possible to decode the TSA picture and thenext picture in decoding order to the TSA picture (TID is greater thanor equal to N).

FIG. 13 is a diagram for describing the TSA picture.

For example, the 14th picture in decoding order is a TSA picture, andthe TID of the picture indicates that TID=N.

In this example, the restriction-target pictures are the 14th, the 15th,the 17th, and the 18th pictures, and the reference-forbidden picture isthe 11th and the 12th pictures. It is assumed, for example, that thedecoder has decoded up to the pictures having TID=N−1. In other words,the 10th picture and the 13th picture have been decoded. Here, the 14thpicture having TID=N does not depend on the 11th picture or the 12thpicture which has not yet been decoded. Accordingly, when it is desiredto decode a picture having TIDs greater than or equal to N, the decoderis capable of starting, from the TSA picture, decoding of the pictureshaving TIDs greater than or equal to N. This means that the decoder iscapable of starting decoding of pictures in not only the layer of TID=Nin which the TSA picture is present but also layers of greater values.In this example, decoding of the 15th picture can also be started.

STSA picture is a picture having a similar meaning to the TSA picture.

STSA picture is a picture that means the following. STSA picture meansthat, when a TID of the STSA picture is N, the STSA picture and apicture which is subsequent in decoding order to the STSA picture andhas a TID of N (restriction-target picture) do not refer to a picturewhich precedes the STSA picture in decoding order and has a TID of N(reference-forbidden picture).

FIG. 14 is a diagram for describing the STSA picture.

In this example, the 14th picture in decoding order is the STSA picture,and the TID of the picture indicates that TID=N. When decoding picturesin the layer of TID=N−1, the decoder is capable of starting, from the14th STSA picture, decoding of pictures in the layer of TID=N. However,since the STSA picture puts reference restriction only on the layer inwhich the STSA picture is present, the decoder cannot start decoding ofthe 15th picture. On the other hand, as to the TSA picture, it ispossible to decode the 15th picture as described above.

The following describes comparison between the TSA picture and the STSApicture.

When a TSA picture is present in the layer of TID=N, it is possible toresume decoding from any one of pictures greater than or equal to TID=Nsubsequent to the TSA picture in decoding order.

On the other hand, when a STSA picture is present in the layer of TID=N,it is possible to resume decoding only from a picture of TID=Nsubsequent to the STSA picture in decoding order. With the STSA picture,compared to the TSA picture, although upswitching is possible by onlyone level, decrease in coding efficiency is small because of less strictreference restriction.

FIG. 15 is a flowchart which illustrates an example of decodingprocessing performed by a decoder 100 which is the image processingapparatus according to this exemplary embodiment. The case where NALunits in each layer lower than or equal to TID=N−1 have been decoded,and decoding of NAL units in each layer higher than or equal to TID=N isto be started, will be described here as an example.

First, the decoder 100 decodes a NAL unit header of a NAL unit which isa target for processing, and obtains a NAL unit type (NALUT) and atemporal ID (TID) (Step S801).

Next, the decoder 100 checks whether or not the NALUT and the TIDsatisfy upswitch conditions (Step S802). More specifically, the decoder100 checks whether or not the NALUT is TSA or STSA. When the NALUT isTSA, the decoder 100 further checks whether or not the TID is greaterthan or equal to N. When the NALUT is STSA, the decoder 100 furtherchecks whether or not the TID is N.

When the upswitch condition that (i) the NALUT is TSA and the TID isgreater than or equal to N or (ii) the NALUT is STSA and the TID is N issatisfied (Yes in Step S802), the decoder 100 decodes a target picturecorresponding to the target NAL unit (Step S803), and a picture of TID=Nwhich is subsequent to the target picture in decoding order is also atarget for decoding (Step S804). In other words, the decoder 100performs layer movement, thereby including, into a decoding target, alayer in which the target picture is present.

It is to be noted that, when the NALUT is TSA and the TID of a pictureon which decoding is performed subsequently to the TSA picture is largerthan N, the decoder 100 includes the layer of the picture into a targetdecoded subsequently.

On the other hand, when the NALUT and the TID included in the NAL unitheader do not satisfy the upswitch conditions (No in Step S802), thedecoder 100 does not perform layer movement, and as a result, does notinclude, into a decoding target, a layer of a target picturecorresponding to the target NAL unit (Step S805). Then, the decoder 100performs determining processing (Step S801 and Step S802), in the samemanner, on a subsequent picture, as necessary.

As described above, it is possible to perform decoding processing whichinvolves upswitching, with a simple configuration.

Here, in the case of the decoding processing exemplified by theflowchart in FIG. 15, there is a possibility that upswitching cannot beperformed due to an error occurring in a bit stream.

In view of the above, the decoder 100 which is an image processingapparatus according to this exemplary embodiment may perform decodingprocessing as illustrated in FIG. 16.

FIG. 16 is a flowchart which illustrates an example of decodingprocessing performed by the decoder 100 which is the image processingapparatus according to this exemplary embodiment. The case where NALunits in each layer lower than or equal to TID=N−1 have been decoded,and decoding of NAL units in each layer higher than or equal to TID=N isto be started, will be described here as an example.

First, the NAL unit syntax analysis unit 103 of the decoder 100 decodesa NAL unit header of a NAL unit which is a target for processing, in thesame manner as in Step S801 illustrated in FIG. 15, and obtains a NALunit type (NALUT) and a temporal ID (TID) (Step S401).

Next, the NAL unit syntax analysis unit 103 checks whether or not theNALUT and the TID satisfy upswitch conditions in the same manner as inStep S802 illustrated in FIG. 15 (Step S102). More specifically, the NALunit syntax analysis unit 103 checks whether or not the NALUT is one ofTSA and STSA. When the NALUT is TSA, the NAL unit syntax analysis unit103 further checks whether or not the TID is greater than or equal to N.When the NALUT is STSA, the NAL unit syntax analysis unit 103 furtherchecks whether or not the TID is N

When the upswitch condition that (i) the NALUT is TSA and the TID isgreater than or equal to N or (ii) the NALUT is STSA and the TID is N issatisfied (Yes in Step S402), the data decoding unit 104 of the decoder100, for each reference picture which is referred to by a target picturecorresponding to the target NAL unit, decodes the number of thereference picture, and checks whether or not a reference picture havingthe number is present in the DPB buffer 105 (Step S411). Then, the datadecoding unit 104 checks whether each of the reference pictures are allpresent in the DPB buffer 105 (Step S412).

When all of the reference pictures are present in the DPB buffer 105(Yes in Step S412), the data decoding unit 104 decodes the targetpicture in the same manner as in Step S803 illustrated in FIG. 15 (StepS403). In addition, the data decoding unit 104 determines that a pictureof TID=N which is subsequent to the target picture in decoding order isalso a target for decoding, in the same manner as in Step S804illustrated in FIG. 15 (Step S404). In other words, the data decodingunit 104 includes a layer in which the target picture is present intothe decoding target. It is to be noted that, when the NALUT is TSA andthe TID of a picture on which decoding is performed subsequently to theTSA picture is larger than N, the data decoding unit 104 includes thelayer of the picture into a target to be decoded subsequently.

On the other hand, when at least one reference picture is not present inthe DPB buffer 105 (No in Step S412), the data decoding unit 104 decodesthe reference picture which is not present (Step S413). For example,when a bit stream (NAL unit) related to the reference picture which isnot present is present in a local memory (the CPB buffer 102, forexample), the data decoding unit 104 decodes an encoded referencepicture in a bit stream stored in the CPB buffer 102. When the decoder100 is receiving a bit stream via a network and an encoded referencepicture is not present in a local memory, the data decoding unit 104 mayrequest a server which is transmitting the bit stream to transmit thereference picture. Alternatively, the data decoding unit 104 may decode,as an alternative of the above-described reference picture, a picturewhich is temporally close to the target picture among decodable bitstreams. Or, the data decoding unit 104 may use, as an alternative of adecoded reference picture, a decoded picture which is stored in the DPBbuffer 105 and is temporally closest to the target picture.

Next, the data decoding unit 104 checks whether or not decoding of areference picture has succeeded (Step S414). When the decoding hassucceeded (Yes in Step S414), the data decoding unit 104 decodes thetarget picture (Step S403). On the other hand, when the data decodingunit 104 fails decoding of even one reference picture (No in Step S414),the data decoding unit 104 does not perform layer movement in the samemanner as in Step S805 illustrated in FIG. 15, and as a result, does notinclude a layer of the target picture corresponding to the target NALunit into the decoding target (Step S405). It is to be noted that, inStep S405, even when an image decoded halfway is present, this image isneither stored in the DPB buffer 105 nor displayed.

In addition, when the NALUT and the TID included in the NAL unit headerdo not satisfy the upswitch condition (No in Step S402), the decoder 100does not perform layer movement for the target picture in the samemanner as above (Step S405).

It is to be noted that, when a picture which satisfies the upswitchcondition is present next in decoding order, upswitching may be retriedfor the picture. When the retry is repeated more than predeterminedtimes, or a waiting interval from trying the upswitching operation to anactual execution exceeds more than a predetermined time, the decoder 100may notify of an error signal to an external means (a control system ora user interface of a system layer) which issued the instruction ofupswitching. The external means which has received the error signal maynotify or display the failure of upswitching using a drawing means.Alternatively, the external means may perform an abnormal time operation(for example, displaying a message on a display screen, causing anapparatus to vibrate, and so on) using a user interface.

As described above, the decoder 100 according to an aspect of thepresent disclosure is capable of providing a method of upswitching, andfurther decoding a bit stream even when an error is occurring in the bitstream and a reference picture is not present in the DPB buffer 105.

In sum, according to this exemplary embodiment, an image signal includesa plurality of NAL units which are signal units, pictures eachcorresponding to a different one of the NAL units are classified intoany one of a plurality of layers, and a target picture is encodedwithout referring to a picture which belongs to a layer in a higherlevel than a layer to which the target picture belongs. In obtaining ofa NAL unit type according to this exemplary embodiment, a NAL unit typeis obtained which is a NAL unit type included in a NAL unitcorresponding to a target picture to be processed among a plurality ofpictures, and which defines forbidding a reference relation in which thetarget picture refers to a specific picture which belongs to the samelayer as the target picture. In the image processing method according tothis exemplary embodiment, when a restriction is imposed on a layer towhich a picture to be decoded among a plurality of pictures belongs, itis further determined whether or not upswitching for alleviating therestriction on the layer can be performed on the target picture, basedon the obtained NAL unit type and the layer to which the target picturebelongs. In executing the restriction alleviation processing accordingto this exemplary embodiment, when it is determined that the upswitchingcan be performed, whether or not the above-described specific picturewhich is a reference picture to which the target picture refers isstored in the DPB buffer 105 is determined. When it is found that thespecific picture is not stored in the DPB buffer 105 as a result of thedetermination, the reference picture is decoded, and the target pictureis decoded by performing referring to the reference picture which isforbidden by the above-described NAL unit type.

There is the case, for example, where a NAL unit type such as TSA orSTSA which forbids a reference relation in which a specific picture isreferred to, in spite of the fact that a target picture refers to thespecific picture, is falsely included in a NAL unit. In such a case,since the specific picture is not stored in the DPB buffer 105, thetarget picture cannot refer to the specific picture in performing ofupswitching, in some cases. In view of the above, when a specificpicture is not stored in the DPB buffer 105 as a reference picture,since the reference picture is decoded and alleviation allows referringto the reference picture which is forbidden by the NAL unit type asdescribed above, it is possible to properly decode the target picture.

Next, the temporal nesting flag (Temporal_Nesting_Flag) will beexplained.

FIG. 17A to FIG. 17D are diagrams for explaining a temporal nestingflag.

As illustrated in FIG. 17A to FIG. 17D, when a temporal nesting flag istrue (Temporal_Nesting_Flag=1), depending on the relation of TID betweentwo arbitrary pictures pic Band pic C which precede a picture pic A indecoding order, reference to the picture pic B by the picture pic A isrestricted.

To be specific, the decoding order between the pictures pic A to pic Cis the picture pic B, the picture pic C, the picture pic A. Asillustrated in FIG. 17A and FIG. 17B, when the temporal nesting flag istrue and the TID of the picture pic C is smaller than the TID of thepicture pic B, restriction is imposed on reference to the picture pic Bwhich precedes the picture pic C in decoding order. It is to be notedthat, even when the temporal nesting flag is true as illustrated in FIG.17C and FIG. 17D, reference to the picture pic B is not restricted whenthe TID of the picture pic C and the TID of the picture pic B are thesame.

Here, there is sometimes the case where an error occurs that a referencestructure between pictures is not restricted in spite of the fact that atemporal nesting flag of true is set. In other words, also in this case,there is the case where, in decoding of a target picture, a referencepicture to which the target picture refers is not present in the DPBbuffer 105.

FIG. 18 is a flowchart which illustrates an example of decodingprocessing performed by the decoder 100 which is the image processingapparatus according to this exemplary embodiment.

First, the data decoding unit 104 of the decoder 100 decodes a videoparameter set (VPS) or a sequence parameter set (SPS) included in a bitstream, and obtains a temporal nesting flag therefrom (Step S501).

Then, the data decoding unit 104 determines whether or not the temporalnesting flag is true (1) (Step S502). Here, when it is determined thatthe temporal nesting flag is true (Yes in Step S502), the data decodingunit 104 decodes, for each of the reference pictures referred to by atarget picture to be processed which corresponds to the temporal nestingflag, the number of the reference picture, and checks whether areference picture having the number is present in the DPB buffer 105(Step S503). Then, the data decoding unit 104 determines, for each ofthe reference pictures, whether the reference pictures are all presentin the DPB buffer 105 (Step S504).

When it is determined that all of the reference pictures are present inthe DPB buffer 105 (Yes in Step S504), the data decoding unit 104decodes the target picture (Step S506). On the other hand, when it isdetermined that at least one reference picture is not present in the DPBbuffer 105 (No in Step S504), the data decoding unit 104 decodes thereference picture which is not present in the DPB buffer 105 in the samemanner as in Step S413 illustrated in FIG. 16. For example, when a bitstream related to the reference picture which is not present is includedin the CPB buffer 102, the data decoding unit 104 decodes an encodedreference picture in the bit stream stored in the CPB buffer 102. Whenthe decoder 100 is receiving a bit stream via a network and the encodedreference picture is not present in the CPB buffer 102, the datadecoding unit 104 may request a server which is transmitting the bitstream to transmit the reference picture. Alternatively, the datadecoding unit 104 may decode, as an alternative of the above-describedreference picture, a picture which is temporally close to the targetpicture among decodable bit streams. Or, the data decoding unit 104 mayuse, as an alternative of a decoded reference picture, a decoded picturewhich is stored in the DPB buffer 105 and is temporally close to thetarget picture.

Next, the data decoding unit 104 decodes a target picture by referringto a reference picture in the DPB buffer 105 or the reference picturewhich is decoded in Step S505 (Step S506).

With the image processing method according to this exemplary embodiment,a target picture can be decoded by referring to a reference picture whenthe reference picture is present in the DPB buffer 105, and even whenthe reference picture is not present in the DPB buffer 105, thereference picture is trace-backed to and decoded, thereby allowingdecoding even in more error cases.

In other words, according to this exemplary embodiment, a plurality ofpictures are each classified into any one of a plurality of layers, andencoded without referring to a picture which belongs to a higher levellayer than a layer to which the picture belongs. Furthermore, accordingto this exemplary embodiment, a temporal nesting flag which forbidsreferring to, in a predetermined state, by a target picture to beprocessed among a plurality of pictures, a lower level picture whichbelongs to a layer in a lower level than a layer to which the targetpicture belongs, is obtained as a parameter. In addition, in executingof the restriction alleviation processing according to this exemplaryembodiment, it is determined whether or not a lower level picture isstored in the DPB buffer 105. Here, when it is determined that the lowerlevel picture is not stored in the DPB buffer 105, the lower levelpicture is decoded, and the target picture is decoded by referring tothe decoded lower level picture without complying with the temporalnesting flag.

There is the case, for example, a temporal nesting which forbidsreferring to a lower level picture in spite of the fact that a targetpicture refers to the lower level picture is falsely included in animage signal. In such a case, the target picture cannot refer to thelower level picture because the lower level picture is not stored in theDPB buffer 105. In view of the above, when a lower level picture is notstored in the DPB buffer 105, since the lower level picture is decodedand alleviation allows referring to the lower level picture which isforbidden by the temporal nesting as described above, it is possible toproperly decode the target picture.

Although the image processing method and image processing apparatusaccording to one or more aspects have been described based on each ofexemplary embodiments and modifications, the present disclosure alsoincludes, for example, other aspects as below.

FIG. 19A is a flowchart which illustrates an image processing methodaccording to an aspect of the present disclosure.

The image processing method is an image processing method for processingan image signal generated by encoding a plurality of pictures, andincludes execution of the processes of Step S11 and Step S12. In StepS11, a parameter which indicates restriction on a reference relationbetween at least one of a plurality of pictures and an other one of theplurality of pictures in encoding is obtained from the image signal.Next, in Step S12, restriction alleviation processing for alleviatingthe restriction on the reference relation indicated by the parameter isperformed.

FIG. 19B is a block diagram which illustrates a configuration of theimage processing apparatus according to an aspect of the presentdisclosure.

An image processing apparatus 10 is an image processing apparatus whichprocesses an image signal generated by encoding a plurality of pictures,and includes a parameter obtaining unit 11 an a processing unit 12.

The parameter obtaining unit 11 obtains, from the image signal, aparameter which indicates restriction on a reference relation between atleast one of a plurality of pictures and an other one of the pluralityof pictures in encoding. The processing unit 12 performs restrictionalleviation processing for alleviating the restriction indicated by theparameter.

With the image processing method and the image processing apparatusaccording to an aspect of the present disclosure as described above, theadvantageous effects equivalent to the above-described Embodiments andmodifications are produced. In other words, with the above-described oneaspect, even when restriction on a reference relation indicated by aparameter is falsely strict because the parameter such as a NAL unittype or a temporal nesting flag is incorrect, the restriction on thereference relation is alleviated. It is therefore possible to prevent anerror from occurring due to inconsistency that occurs between therestriction on the reference relation indicated by the parameter and anactual reference relation between image signals (relation indicated bythe number of a reference picture, or the like) or a reference relationindicated by another parameter (different NAL unit type). Thus, it ispossible to prevent decoding on an image signal from ending due to anerror, for example.

Each of the structural elements in each of the above-describedembodiments and modifications may be configured in the form of anexclusive hardware product, or may be realized by executing a softwareprogram suitable for the structural element. Each of the structuralelements may be realized by means of a program executing unit, such as aCPU and a processor, reading and executing the software program recordedon a recording medium such as a hard disk or a semiconductor memory.Here, the software for realizing the image processing apparatus or thelike according to each of the embodiments and modifications is a programwhich executes the processing of each of the steps in the flowchartillustrated in FIG. 19A. Although the image processing method and imageprocessing apparatus according to one or more aspects have beendescribed based on each of exemplary embodiments and modifications, thescope of the present disclosure is not limited to these embodiments andmodifications. Those skilled in the art will readily appreciate thatvarious modifications may be made in these exemplary embodiments andthat other embodiments may be obtained by arbitrarily combining thestructural elements of the embodiments without materially departing fromthe novel teachings and advantages of the subject matter recited in theappended Claims. Accordingly, all such modifications and otherembodiments are included in the present disclosure.

Embodiment 4

The processing described in each of embodiments can be simplyimplemented in an independent computer system, by recording, in arecording medium, one or more programs for implementing theconfigurations of the moving picture encoding method (image encodingmethod) and the moving picture decoding method (image decoding method,image processing method) described in each of embodiments. The recordingmedia may be any recording media as long as the program can be recorded,such as a magnetic disk, an optical disk, a magnetic optical disk, an ICcard, and a semiconductor memory.

Hereinafter, the applications to the moving picture encoding method(image encoding method) and the moving picture decoding method (imagedecoding method, image processing method) described in each ofembodiments and systems using thereof will be described. The system hasa feature of having an image coding apparatus that includes an imageencoding apparatus using the image encoding method and an image decodingapparatus (image processing apparatus) using the image decoding method(image processing method). Other configurations in the system can bechanged as appropriate depending on the cases.

FIG. 20 illustrates an overall configuration of a content providingsystem ex100 for implementing content distribution services. The areafor providing communication services is divided into cells of desiredsize, and base stations ex106, ex107, ex108, ex109, and ex110 which arefixed wireless stations are placed in each of the cells.

The content providing system ex100 is connected to devices, such as acomputer ex111, a personal digital assistant (PDA) ex112, a cameraex113, a cellular phone ex114 and a game machine ex115, via the Internetex101, an Internet service provider ex102, a telephone network ex104, aswell as the base stations ex106 to ex110, respectively.

However, the configuration of the content providing system ex100 is notlimited to the configuration shown in FIG. 20, and a combination inwhich any of the elements are connected is acceptable. In addition, eachdevice may be directly connected to the telephone network ex104, ratherthan via the base stations ex106 to ex110 which are the fixed wirelessstations. Furthermore, the devices may be interconnected to each othervia a short distance wireless communication and others.

The camera ex113, such as a digital video camera, is capable ofcapturing video. A camera ex116, such as a digital camera, is capable ofcapturing both still images and video. Furthermore, the cellular phoneex114 may be the one that meets any of the standards such as GlobalSystem for Mobile Communications (GSM) (registered trademark), CodeDivision Multiple Access (CDMA), Wideband-Code Division Multiple Access(W-CDMA), Long Term Evolution (LTE), and High Speed Packet Access(HSPA). Alternatively, the cellular phone ex114 may be a PersonalHandyphone System (PHS).

In the content providing system ex100, a streaming server ex103 isconnected to the camera ex113 and others via the telephone network ex104and the base station ex109, which enables distribution of images of alive show and others. In such a distribution, a content (for example,video of a music live show) captured by the user using the camera ex113is encoded as described above in each of embodiments (i.e., the camerafunctions as the image encoding apparatus according to an aspect of thepresent disclosure), and the encoded content is transmitted to thestreaming server ex103. On the other hand, the streaming server ex103carries out stream distribution of the transmitted content data to theclients upon their requests. The clients include the computer ex111, thePDA ex112, the camera ex113, the cellular phone ex114, and the gamemachine ex115 that are capable of decoding the above-mentioned encodeddata. Each of the devices that have received the distributed datadecodes and reproduces the encoded data (i.e., functions as the imageprocessing apparatus or the image decoding apparatus according to anaspect of the present disclosure).

The captured data may be encoded by the camera ex113 or the streamingserver ex103 that transmits the data, or the encoding processes may beshared between the camera ex113 and the streaming server ex103.Similarly, the distributed data may be decoded by the clients or thestreaming server ex103, or the decoding processes may be shared betweenthe clients and the streaming server ex103. Furthermore, the data of thestill images and video captured by not only the camera ex113 but alsothe camera ex116 may be transmitted to the streaming server ex103through the computer ex111. The encoding processes may be performed bythe camera ex116, the computer ex111, or the streaming server ex103, orshared among them.

Furthermore, the coding processes may be performed by an LSI ex500generally included in each of the computer ex111 and the devices. TheLSI ex500 may be configured of a single chip or a plurality of chips.Software for coding video may be integrated into some type of arecording medium (such as a CD-ROM, a flexible disk, and a hard disk)that is readable by the computer ex111 and others, and the codingprocesses may be performed using the software. Furthermore, when thecellular phone ex114 is equipped with a camera, the video data obtainedby the camera may be transmitted. The video data is data encoded by theLSI ex500 included in the cellular phone ex114.

Furthermore, the streaming server ex103 may be composed of servers andcomputers, and may decentralize data and process the decentralized data,record, or distribute data.

As described above, the clients may receive and reproduce the encodeddata in the content providing system ex100. In other words, the clientscan receive and decode information transmitted by the user, andreproduce the decoded data in real time in the content providing systemex100, so that the user who does not have any particular right andequipment can implement personal broadcasting.

Aside from the example of the content providing system ex100, at leastone of the moving picture coding apparatus (image coding apparatus)described in each of embodiments may be implemented in a digitalbroadcasting system ex200 illustrated in FIG. 21. More specifically, abroadcast station ex201 communicates or transmits, via radio waves to abroadcast satellite ex202, multiplexed data obtained by multiplexingaudio data and others onto video data. The video data is data encoded bythe moving picture encoding method described in each of embodiments(i.e., data encoded by the image encoding apparatus according to anaspect of the present disclosure). Upon receipt of the multiplexed data,the broadcast satellite ex202 transmits radio waves for broadcasting.Then, a home-use antenna ex204 with a satellite broadcast receptionfunction receives the radio waves. Next, a device such as a television(receiver) ex300 and a set top box (STB) ex217 decodes the receivedmultiplexed data, and reproduces the decoded data (i.e., functions asthe image decoding apparatus, image processing apparatus according to anaspect of the present disclosure).

Furthermore, a reader/recorder ex218 (i) reads and decodes themultiplexed data recorded on a recording medium ex215, such as a DVD anda BD, or (i) encodes video signals in the recording medium ex215, and insome cases, writes data obtained by multiplexing an audio signal on theencoded data. The reader/recorder ex218 can include the moving picturedecoding apparatus or the moving picture encoding apparatus as shown ineach of embodiments. In this case, the reproduced video signals aredisplayed on the monitor ex219, and can be reproduced by another deviceor system using the recording medium ex215 on which the multiplexed datais recorded. It is also possible to implement the moving picturedecoding apparatus in the set top box ex217 connected to the cable ex203for a cable television or to the antenna ex204 for satellite and/orterrestrial broadcasting, so as to display the video signals on themonitor ex219 of the television ex300. The moving picture decodingapparatus may be implemented not in the set top box but in thetelevision ex300.

FIG. 22 illustrates the television (receiver) ex300 that uses the movingpicture encoding method and the moving picture decoding method describedin each of embodiments. The television ex300 includes: a tuner ex301that obtains or provides multiplexed data obtained by multiplexing audiodata onto video data, through the antenna ex204 or the cable ex203, etc.that receives a broadcast; a modulation/demodulation unit ex302 thatdemodulates the received multiplexed data or modulates data intomultiplexed data to be supplied outside; and amultiplexing/demultiplexing unit ex303 that demultiplexes the modulatedmultiplexed data into video data and audio data, or multiplexes videodata and audio data encoded by a signal processing unit ex306 into data.

The television ex300 further includes: a signal processing unit ex306including an audio signal processing unit ex304 and a video signalprocessing unit ex305 that code each of audio data and video data,(which function as the image coding apparatus according to the aspectsof the present disclosure); and an output unit ex309 including a speakerex307 that provides the decoded audio signal, and a display unit ex308that displays the decoded video signal, such as a display. Furthermore,the television ex300 includes an interface unit ex317 including anoperation input unit ex312 that receives an input of a user operation.Furthermore, the television ex300 includes a control unit ex310 thatcontrols overall each constituent element of the television ex300, and apower supply circuit unit ex311 that supplies power to each of theelements. Other than the operation input unit ex312, the interface unitex317 may include: a bridge ex313 that is connected to an externaldevice, such as the reader/recorder ex218; a slot unit ex314 forenabling attachment of the recording medium ex216, such as an SD card; adriver ex315 to be connected to an external recording medium, such as ahard disk; and a modem ex316 to be connected to a telephone network.Here, the recording medium ex216 can electrically record informationusing a non-volatile/volatile semiconductor memory element for storage.The constituent elements of the television ex300 are connected to eachother through a synchronous bus.

First, the configuration in which the television ex300 decodesmultiplexed data obtained from outside through the antenna ex204 andothers and reproduces the decoded data will be described. In thetelevision ex300, upon a user operation through a remote controllerex220 and others, the multiplexing/demultiplexing unit ex303demultiplexes the multiplexed data demodulated by themodulation/demodulation unit ex302, under control of the control unitex310 including a CPU. Furthermore, the audio signal processing unitex304 decodes the demultiplexed audio data, and the video signalprocessing unit ex305 decodes the demultiplexed video data, using thedecoding method described in each of embodiments, in the televisionex300. The output unit ex309 provides the decoded video signal and audiosignal outside, respectively. When the output unit ex309 provides thevideo signal and the audio signal, the signals may be temporarily storedin buffers ex318 and ex319, and others so that the signals arereproduced in synchronization with each other. Furthermore, thetelevision ex300 may read multiplexed data not through a broadcast andothers but from the recording media ex215 and ex216, such as a magneticdisk, an optical disk, and a SD card. Next, a configuration in which thetelevision ex300 encodes an audio signal and a video signal, andtransmits the data outside or writes the data on a recording medium willbe described. In the television ex300, upon a user operation through theremote controller ex220 and others, the audio signal processing unitex304 encodes an audio signal, and the video signal processing unitex305 encodes a video signal, under control of the control unit ex310using the encoding method described in each of embodiments. Themultiplexing/demultiplexing unit ex303 multiplexes the encoded videosignal and audio signal, and provides the resulting signal outside. Whenthe multiplexing/demultiplexing unit ex303 multiplexes the video signaland the audio signal, the signals may be temporarily stored in thebuffers ex320 and ex321, and others so that the signals are reproducedin synchronization with each other. Here, the buffers ex318, ex319,ex320, and ex321 may be plural as illustrated, or at least one buffermay be shared in the television ex300. Furthermore, data may be storedin a buffer so that the system overflow and underflow may be avoidedbetween the modulation/demodulation unit ex302 and themultiplexing/demultiplexing unit ex303, for example.

Furthermore, the television ex300 may include a configuration forreceiving an AV input from a microphone or a camera other than theconfiguration for obtaining audio and video data from a broadcast or arecording medium, and may encode the obtained data. Although thetelevision ex300 can encode, multiplex, and provide outside data in thedescription, it may be capable of only receiving, decoding, andproviding outside data but not the encoding, multiplexing, and providingoutside data.

Furthermore, when the reader/recorder ex218 reads or writes multiplexeddata from or on a recording medium, one of the television ex300 and thereader/recorder ex218 may code the multiplexed data, and the televisionex300 and the reader/recorder ex218 may share the coding partly.

As an example, FIG. 23 illustrates a configuration of an informationreproducing/recording unit ex400 when data is read or written from or onan optical disk. The information reproducing/recording unit ex400includes constituent elements ex401, ex402, ex403, ex404, ex405, ex406,and ex407 to be described hereinafter. The optical head ex401 irradiatesa laser spot in a recording surface of the recording medium ex215 thatis an optical disk to write information, and detects reflected lightfrom the recording surface of the recording medium ex215 to read theinformation. The modulation recording unit ex402 electrically drives asemiconductor laser included in the optical head ex401, and modulatesthe laser light according to recorded data. The reproductiondemodulating unit ex403 amplifies a reproduction signal obtained byelectrically detecting the reflected light from the recording surfaceusing a photo detector included in the optical head ex401, anddemodulates the reproduction signal by separating a signal componentrecorded on the recording medium ex215 to reproduce the necessaryinformation. The buffer ex404 temporarily holds the information to berecorded on the recording medium ex215 and the information reproducedfrom the recording medium ex215. The disk motor ex405 rotates therecording medium ex215. The servo control unit ex406 moves the opticalhead ex401 to a predetermined information track while controlling therotation drive of the disk motor ex405 so as to follow the laser spot.The system control unit ex407 controls overall the informationreproducing/recording unit ex400. The reading and writing processes canbe implemented by the system control unit ex407 using variousinformation stored in the buffer ex404 and generating and adding newinformation as necessary, and by the modulation recording unit ex402,the reproduction demodulating unit ex403, and the servo control unitex406 that record and reproduce information through the optical headex401 while being operated in a coordinated manner. The system controlunit ex407 includes, for example, a microprocessor, and executesprocessing by causing a computer to execute a program for read andwrite.

Although the optical head ex401 irradiates a laser spot in thedescription, it may perform high-density recording using near fieldlight.

FIG. 24 illustrates the recording medium ex215 that is the optical disk.On the recording surface of the recording medium ex215, guide groovesare spirally formed, and an information track ex230 records, in advance,address information indicating an absolute position on the diskaccording to change in a shape of the guide grooves. The addressinformation includes information for determining positions of recordingblocks ex231 that are a unit for recording data. Reproducing theinformation track ex230 and reading the address information in anapparatus that records and reproduces data can lead to determination ofthe positions of the recording blocks. Furthermore, the recording mediumex215 includes a data recording area ex233, an inner circumference areaex232, and an outer circumference area ex234. The data recording areaex233 is an area for use in recording the user data. The innercircumference area ex232 and the outer circumference area ex234 that areinside and outside of the data recording area ex233, respectively arefor specific use except for recording the user data. The informationreproducing/recording unit 400 reads and writes encoded audio, encodedvideo data, or multiplexed data obtained by multiplexing the encodedaudio and video data, from and on the data recording area ex233 of therecording medium ex215.

Although an optical disk having a layer, such as a DVD and a BD isdescribed as an example in the description, the optical disk is notlimited to such, and may be an optical disk having a multilayerstructure and capable of being recorded on a part other than thesurface. Furthermore, the optical disk may have a structure formultidimensional recording/reproduction, such as recording ofinformation using light of colors with different wavelengths in the sameportion of the optical disk and for recording information havingdifferent layers from various angles.

Furthermore, a car ex210 having an antenna ex205 can receive data fromthe satellite ex202 and others, and reproduce video on a display devicesuch as a car navigation system ex211 set in the car ex210, in thedigital broadcasting system ex200. Here, a configuration of the carnavigation system ex211 will be a configuration, for example, includinga GPS receiving unit from the configuration illustrated in FIG. 22. Thesame will be true for the configuration of the computer ex111, thecellular phone ex114, and others.

FIG. 25A illustrates the cellular phone ex114 that uses the movingpicture coding method described in embodiments. The cellular phone ex114includes: an antenna ex350 for transmitting and receiving radio wavesthrough the base station ex110; a camera unit ex365 capable of capturingmoving and still images; and a display unit ex358 such as a liquidcrystal display for displaying the data such as decoded video capturedby the camera unit ex365 or received by the antenna ex350. The cellularphone ex114 further includes: a main body unit including an operationkey unit ex366; an audio output unit ex357 such as a speaker for outputof audio; an audio input unit ex356 such as a microphone for input ofaudio; a memory unit ex367 for storing captured video or still pictures,recorded audio, coded data of the received video, the still pictures,e-mails, or others; and a slot unit ex364 that is an interface unit fora recording medium that stores data in the same manner as the memoryunit ex367.

Next, an example of a configuration of the cellular phone ex114 will bedescribed with reference to FIG. 25B. In the cellular phone ex114, amain control unit ex360 designed to control overall each unit of themain body including the display unit ex358 as well as the operation keyunit ex366 is connected mutually, via a synchronous bus ex370, to apower supply circuit unit ex361, an operation input control unit ex362,a video signal processing unit ex355, a camera interface unit ex363, aliquid crystal display (LCD) control unit ex359, amodulation/demodulation unit ex352, a multiplexing/demultiplexing unitex353, an audio signal processing unit ex354, the slot unit ex364, andthe memory unit ex367.

When a call-end key or a power key is turned ON by a user's operation,the power supply circuit unit ex361 supplies the respective units withpower from a battery pack so as to activate the cell phone ex114.

In the cellular phone ex114, the audio signal processing unit ex354converts the audio signals collected by the audio input unit ex356 invoice conversation mode into digital audio signals under the control ofthe main control unit ex360 including a CPU, ROM, and RAM. Then, themodulation/demodulation unit ex352 performs spread spectrum processingon the digital audio signals, and the transmitting and receiving unitex351 performs digital-to-analog conversion and frequency conversion onthe data, so as to transmit the resulting data via the antenna ex350.Also, in the cellular phone ex114, the transmitting and receiving unitex351 amplifies the data received by the antenna ex350 in voiceconversation mode and performs frequency conversion and theanalog-to-digital conversion on the data. Then, themodulation/demodulation unit ex352 performs inverse spread spectrumprocessing on the data, and the audio signal processing unit ex354converts it into analog audio signals, so as to output them via theaudio output unit ex357.

Furthermore, when an e-mail in data communication mode is transmitted,text data of the e-mail inputted by operating the operation key unitex366 and others of the main body is sent out to the main control unitex360 via the operation input control unit ex362. The main control unitex360 causes the modulation/demodulation unit ex352 to perform spreadspectrum processing on the text data, and the transmitting and receivingunit ex351 performs the digital-to-analog conversion and the frequencyconversion on the resulting data to transmit the data to the basestation ex110 via the antenna ex350. When an e-mail is received,processing that is approximately inverse to the processing fortransmitting an e-mail is performed on the received data, and theresulting data is provided to the display unit ex358.

When video, still images, or video and audio in data communication modeis or are transmitted, the video signal processing unit ex355 compressesand encodes video signals supplied from the camera unit ex365 using themoving picture encoding method shown in each of embodiments (i.e.,functions as the image encoding apparatus according to the aspect of thepresent disclosure), and transmits the encoded video data to themultiplexing/demultiplexing unit ex353. In contrast, during when thecamera unit ex365 captures video, still images, and others, the audiosignal processing unit ex354 encodes audio signals collected by theaudio input unit ex356, and transmits the encoded audio data to themultiplexing/demultiplexing unit ex353.

The multiplexing/demultiplexing unit ex353 multiplexes the encoded videodata supplied from the video signal processing unit ex355 and theencoded audio data supplied from the audio signal processing unit ex354,using a predetermined method. Then, the modulation/demodulation unit(modulation/demodulation circuit unit) ex352 performs spread spectrumprocessing on the multiplexed data, and the transmitting and receivingunit ex351 performs digital-to-analog conversion and frequencyconversion on the data so as to transmit the resulting data via theantenna ex350.

When receiving data of a video file which is linked to a Web page andothers in data communication mode or when receiving an e-mail with videoand/or audio attached, in order to decode the multiplexed data receivedvia the antenna ex350, the multiplexing/demultiplexing unit ex353demultiplexes the multiplexed data into a video data bit stream and anaudio data bit stream, and supplies the video signal processing unitex355 with the encoded video data and the audio signal processing unitex354 with the encoded audio data, through the synchronous bus ex370.The video signal processing unit ex355 decodes the video signal using amoving picture decoding method corresponding to the moving pictureencoding method shown in each of embodiments (i.e., functions as theimage processing apparatus or image decoding apparatus according to theaspect of the present disclosure), and then the display unit ex358displays, for instance, the video and still images included in the videofile linked to the Web page via the LCD control unit ex359. Furthermore,the audio signal processing unit ex354 decodes the audio signal, and theaudio output unit ex357 provides the audio.

Furthermore, similarly to the television ex300, a terminal such as thecellular phone ex114 probably have 3 types of implementationconfigurations including not only (i) a transmitting and receivingterminal including both an encoding apparatus and a decoding apparatus,but also (ii) a transmitting terminal including only an encodingapparatus and (iii) a receiving terminal including only a decodingapparatus. Although the digital broadcasting system ex200 receives andtransmits the multiplexed data obtained by multiplexing audio data ontovideo data in the description, the multiplexed data may be data obtainedby multiplexing not audio data but character data related to video ontovideo data, and may be not multiplexed data but video data itself.

As such, the moving picture coding method in each of embodiments can beused in any of the devices and systems described. Thus, the advantagesdescribed in each of embodiments can be obtained.

Furthermore, various modifications and revisions can be made in any ofthe embodiments in the present disclosure.

Furthermore, the present disclosure is not limited to the aforementionedEmbodiments, and various modifications and revisions are possiblewithout departing from the scope of the present disclosure.

Embodiment 5

Video data can be generated by switching, as necessary, between (i) themoving picture encoding method or the moving picture encoding apparatusshown in each of embodiments and (ii) a moving picture encoding methodor a moving picture encoding apparatus in conformity with a differentstandard, such as MPEG-2, MPEG-4 AVC, and VC-1.

Here, when a plurality of video data that conforms to the differentstandards is generated and is then decoded, the decoding methods need tobe selected to conform to the different standards. However, since towhich standard each of the plurality of the video data to be decodedconform cannot be detected, an appropriate decoding method cannot beselected.

In view of this, multiplexed data obtained by multiplexing audio dataand others onto video data has a structure including identificationinformation indicating to which standard the video data conforms. Thespecific structure of the multiplexed data including the video datagenerated in the moving picture encoding method and by the movingpicture encoding apparatus shown in each of embodiments will behereinafter described. The multiplexed data is a digital stream in theMPEG-2 Transport Stream format.

FIG. 26 illustrates a structure of the multiplexed data. As illustratedin FIG. 26, the multiplexed data can be obtained by multiplexing atleast one of a video stream, an audio stream, a presentation graphicsstream (PG), and an interactive graphics stream. The video streamrepresents primary video and secondary video of a movie, the audiostream (IG) represents a primary audio part and a secondary audio partto be mixed with the primary audio part, and the presentation graphicsstream represents subtitles of the movie. Here, the primary video isnormal video to be displayed on a screen, and the secondary video isvideo to be displayed on a smaller window in the primary video.Furthermore, the interactive graphics stream represents an interactivescreen to be generated by arranging the GUI components on a screen. Thevideo stream is encoded in the moving picture encoding method or by themoving picture encoding apparatus shown in each of embodiments, or in amoving picture encoding method or by a moving picture encoding apparatusin conformity with a conventional standard, such as MPEG-2, MPEG-4 AVC,and VC-1. The audio stream is encoded in accordance with a standard,such as Dolby-AC-3, Dolby Digital Plus, MLP, DTS, DTS-HD, and linearPCM.

Each stream included in the multiplexed data is identified by PID. Forexample, 0x1011 is allocated to the video stream to be used for video ofa movie, 0x1100 to 0x111F are allocated to the audio streams, 0x1200 to0x121F are allocated to the presentation graphics streams, 0x1400 to0x141F are allocated to the interactive graphics streams, 0x1B00 to0x1B1F are allocated to the video streams to be used for secondary videoof the movie, and 0x1A00 to 0x1A1F are allocated to the audio streams tobe used for the secondary audio to be mixed with the primary audio.

FIG. 27 schematically illustrates how data is multiplexed. First, avideo stream ex235 composed of video frames and an audio stream ex238composed of audio frames are transformed into a stream of PES packetsex236 and a stream of PES packets ex239, and further into TS packetsex237 and TS packets ex240, respectively. Similarly, data of apresentation graphics stream ex241 and data of an interactive graphicsstream ex244 are transformed into a stream of PES packets ex242 and astream of PES packets ex245, and further into TS packets ex243 and TSpackets ex246, respectively. These TS packets are multiplexed into astream to obtain multiplexed data ex247.

FIG. 28 illustrates how a video stream is stored in a stream of PESpackets in more detail. The first bar in FIG. 28 shows a video framestream in a video stream. The second bar shows the stream of PESpackets. As indicated by arrows denoted as yy1, yy2, yy3, and yy4 inFIG. 28, the video stream is divided into pictures as I pictures, Bpictures, and P pictures each of which is a video presentation unit, andthe pictures are stored in a payload of each of the PES packets. Each ofthe PES packets has a PES header, and the PES header stores aPresentation Time-Stamp (PTS) indicating a display time of the picture,and a Decoding Time-Stamp (DTS) indicating a decoding time of thepicture.

FIG. 29 illustrates a format of TS packets to be finally written on themultiplexed data. Each of the TS packets is a 188-byte fixed lengthpacket including a 4-byte TS header having information, such as a PIDfor identifying a stream and a 184-byte TS payload for storing data. ThePES packets are divided, and stored in the TS payloads, respectively.When a BD ROM is used, each of the TS packets is given a 4-byteTP_Extra_Header, thus resulting in 192-byte source packets. The sourcepackets are written on the multiplexed data. The TP_Extra_Header storesinformation such as an Arrival_Time_Stamp (ATS). The ATS shows atransfer start time at which each of the TS packets is to be transferredto a PID filter. The source packets are arranged in the multiplexed dataas shown at the bottom of FIG. 29. The numbers incrementing from thehead of the multiplexed data are called source packet numbers (SPNs).

Each of the TS packets included in the multiplexed data includes notonly streams of audio, video, subtitles and others, but also a ProgramAssociation Table (PAT), a Program Map Table (PMT), and a Program ClockReference (PCR). The PAT shows what a PID in a PMT used in themultiplexed data indicates, and a PID of the PAT itself is registered aszero. The PMT stores PIDs of the streams of video, audio, subtitles andothers included in the multiplexed data, and attribute information ofthe streams corresponding to the PIDs. The PMT also has variousdescriptors relating to the multiplexed data. The descriptors haveinformation such as copy control information showing whether copying ofthe multiplexed data is permitted or not. The PCR stores STC timeinformation corresponding to an ATS showing when the PCR packet istransferred to a decoder, in order to achieve synchronization between anArrival Time Clock (ATC) that is a time axis of ATSs, and an System TimeClock (STC) that is a time axis of PTSs and DTSs.

FIG. 30 illustrates the data structure of the PMT in detail. A PMTheader is disposed at the top of the PMT. The PMT header describes thelength of data included in the PMT and others. A plurality ofdescriptors relating to the multiplexed data is disposed after the PMTheader. Information such as the copy control information is described inthe descriptors. After the descriptors, a plurality of pieces of streaminformation relating to the streams included in the multiplexed data isdisposed. Each piece of stream information includes stream descriptorseach describing information, such as a stream type for identifying acompression codec of a stream, a stream PID, and stream attributeinformation (such as a frame rate or an aspect ratio). The streamdescriptors are equal in number to the number of streams in themultiplexed data.

When the multiplexed data is recorded on a recording medium and others,it is recorded together with multiplexed data information files.

Each of the multiplexed data information files is management informationof the multiplexed data as shown in FIG. 31. The multiplexed datainformation files are in one to one correspondence with the multiplexeddata, and each of the files includes multiplexed data information,stream attribute information, and an entry map.

As illustrated in FIG. 31, the multiplexed data information includes asystem rate, a reproduction start time, and a reproduction end time. Thesystem rate indicates the maximum transfer rate at which a system targetdecoder to be described later transfers the multiplexed data to a PIDfilter. The intervals of the ATSs included in the multiplexed data areset to not higher than a system rate. The reproduction start timeindicates a PTS in a video frame at the head of the multiplexed data. Aninterval of one frame is added to a PTS in a video frame at the end ofthe multiplexed data, and the PTS is set to the reproduction end time.

As shown in FIG. 32, a piece of attribute information is registered inthe stream attribute information, for each PID of each stream includedin the multiplexed data. Each piece of attribute information hasdifferent information depending on whether the corresponding stream is avideo stream, an audio stream, a presentation graphics stream, or aninteractive graphics stream. Each piece of video stream attributeinformation carries information including what kind of compression codecis used for compressing the video stream, and the resolution, aspectratio and frame rate of the pieces of picture data that is included inthe video stream. Each piece of audio stream attribute informationcarries information including what kind of compression codec is used forcompressing the audio stream, how many channels are included in theaudio stream, which language the audio stream supports, and how high thesampling frequency is. The video stream attribute information and theaudio stream attribute information are used for initialization of adecoder before the player plays back the information.

In the present embodiment, the multiplexed data to be used is of astream type included in the PMT. Furthermore, when the multiplexed datais recorded on a recording medium, the video stream attributeinformation included in the multiplexed data information is used. Morespecifically, the moving picture encoding method or the moving pictureencoding apparatus described in each of embodiments includes a step or aunit for allocating unique information indicating video data generatedby the moving picture encoding method or the moving picture encodingapparatus in each of embodiments, to the stream type included in the PMTor the video stream attribute information. With the configuration, thevideo data generated by the moving picture encoding method or the movingpicture encoding apparatus described in each of embodiments can bedistinguished from video data that conforms to another standard.

Furthermore, FIG. 33 illustrates steps of the moving picture decodingmethod according to the present embodiment. In Step exS100, the streamtype included in the PMT or the video stream attribute informationincluded in the multiplexed data information is obtained from themultiplexed data. Next, in Step exS101, it is determined whether or notthe stream type or the video stream attribute information indicates thatthe multiplexed data is generated by the moving picture encoding methodor the moving picture encoding apparatus in each of embodiments. When itis determined that the stream type or the video stream attributeinformation indicates that the multiplexed data is generated by themoving picture encoding method or the moving picture encoding apparatusin each of embodiments, in Step exS102, decoding is performed by themoving picture decoding method in each of embodiments. Furthermore, whenthe stream type or the video stream attribute information indicatesconformance to the conventional standards, such as MPEG-2, MPEG-4 AVC,and VC-1, in Step exS103, decoding is performed by a moving picturedecoding method in conformity with the conventional standards.

As such, allocating a new unique value to the stream type or the videostream attribute information enables determination whether or not themoving picture decoding method or the moving picture decoding apparatusthat is described in each of embodiments can perform decoding. Even whenmultiplexed data that conforms to a different standard is input, anappropriate decoding method or apparatus can be selected. Thus, itbecomes possible to decode information without any error. Furthermore,the moving picture encoding method or apparatus, or the moving picturedecoding method or apparatus in the present embodiment can be used inthe devices and systems described above.

Embodiment 6

Each of the moving picture coding method and the moving picture codingapparatus in each of embodiments is typically achieved in the form of anintegrated circuit or a Large Scale Integrated (LSI) circuit. As anexample of the LSI, FIG. 34 illustrates a configuration of the LSI ex500that is made into one chip. The LSI ex500 includes elements ex501,ex502, ex503, ex504, ex505, ex506, ex507, ex508, and ex509 to bedescribed below, and the elements are connected to each other through abus ex510. The power supply circuit unit ex505 is activated by supplyingeach of the elements with power when the power supply circuit unit ex505is turned on.

For example, when encoding is performed, the LSI ex500 receives an AVsignal from a microphone ex117, a camera ex113, and others through an AVIO ex509 under control of a control unit ex501 including a CPU ex502, amemory controller ex503, a stream controller ex504, and a drivingfrequency control unit ex512. The received AV signal is temporarilystored in an external memory ex511, such as an SDRAM. Under control ofthe control unit ex501, the stored data is segmented into data portionsaccording to the processing amount and speed to be transmitted to asignal processing unit ex507. Then, the signal processing unit ex507encodes an audio signal and/or a video signal. Here, the encoding of thevideo signal is the encoding described in each of embodiments.Furthermore, the signal processing unit ex507 sometimes multiplexes theencoded audio data and the encoded video data, and a stream IO ex506provides the multiplexed data outside. The provided multiplexed data istransmitted to the base station ex107, or written on the recordingmedium ex215. When data sets are multiplexed, the data should betemporarily stored in the buffer ex508 so that the data sets aresynchronized with each other.

Although the memory ex511 is an element outside the LSI ex500, it may beincluded in the LSI ex500. The buffer ex508 is not limited to onebuffer, but may be composed of buffers. Furthermore, the LSI ex500 maybe made into one chip or a plurality of chips.

Furthermore, although the control unit ex501 includes the CPU ex502, thememory controller ex503, the stream controller ex504, the drivingfrequency control unit ex512, the configuration of the control unitex501 is not limited to such. For example, the signal processing unitex507 may further include a CPU. Inclusion of another CPU in the signalprocessing unit ex507 can improve the processing speed. Furthermore, asanother example, the CPU ex502 may serve as or be a part of the signalprocessing unit ex507, and, for example, may include an audio signalprocessing unit. In such a case, the control unit ex501 includes thesignal processing unit ex507 or the CPU ex502 including a part of thesignal processing unit ex507.

The name used here is LSI, but it may also be called IC, system LSI,super LSI, or ultra LSI depending on the degree of integration.

Moreover, ways to achieve integration are not limited to the LSI, and aspecial circuit or a general purpose processor and so forth can alsoachieve the integration. Field Programmable Gate Array (FPGA) that canbe programmed after manufacturing LSIs or a reconfigurable processorthat allows re-configuration of the connection or configuration of anLSI can be used for the same purpose. Such a programmable logic devicecan typically execute the moving picture coding method according to anyof the above embodiments, by loading or reading from a memory or thelike one or more programs that are included in software or firmware.

In the future, with advancement in semiconductor technology, a brand-newtechnology may replace LSI. The functional blocks can be integratedusing such a technology. The possibility is that the present disclosureis applied to biotechnology.

Embodiment 7

When video data generated in the moving picture encoding method or bythe moving picture encoding apparatus described in each of embodimentsis decoded, compared to when video data that conforms to a conventionalstandard, such as MPEG-2, MPEG-4 AVC, and VC-1 is decoded, theprocessing amount probably increases. Thus, the LSI ex500 needs to beset to a driving frequency higher than that of the CPU ex502 to be usedwhen video data in conformity with the conventional standard is decoded.However, when the driving frequency is set higher, the power consumptionincreases.

In view of this, the moving picture decoding apparatus, such as thetelevision ex300 and the LSI ex500 is configured to determine to whichstandard the video data conforms, and switch between the drivingfrequencies according to the determined standard. FIG. 35 illustrates aconfiguration ex800 in the present embodiment. A driving frequencyswitching unit ex803 sets a driving frequency to a higher drivingfrequency when video data is generated by the moving picture encodingmethod or the moving picture encoding apparatus described in each ofembodiments. Then, the driving frequency switching unit ex803 instructsa decoding processing unit ex801 that executes the moving picturedecoding method described in each of embodiments to decode the videodata. When the video data conforms to the conventional standard, thedriving frequency switching unit ex803 sets a driving frequency to alower driving frequency than that of the video data generated by themoving picture encoding method or the moving picture encoding apparatusdescribed in each of embodiments. Then, the driving frequency switchingunit ex803 instructs the decoding processing unit ex802 that conforms tothe conventional standard to decode the video data.

More specifically, the driving frequency switching unit ex803 includesthe CPU ex502 and the driving frequency control unit ex512 in FIG. 34.Here, each of the decoding processing unit ex801 that executes themoving picture decoding method described in each of embodiments and thedecoding processing unit ex802 that conforms to the conventionalstandard corresponds to the signal processing unit ex507 in FIG. 34. TheCPU ex502 determines to which standard the video data conforms. Then,the driving frequency control unit ex512 determines a driving frequencybased on a signal from the CPU ex502. Furthermore, the signal processingunit ex507 decodes the video data based on the signal from the CPUex502. For example, the identification information described inEmbodiment 5 is probably used for identifying the video data. Theidentification information is not limited to the one described inEmbodiment 5 but may be any information as long as the informationindicates to which standard the video data conforms. For example, whenwhich standard video data conforms to can be determined based on anexternal signal for determining that the video data is used for atelevision or a disk, etc., the determination may be made based on suchan external signal. Furthermore, the CPU ex502 selects a drivingfrequency based on, for example, a look-up table in which the standardsof the video data are associated with the driving frequencies as shownin FIG. 37. The driving frequency can be selected by storing the look-uptable in the buffer ex508 and in an internal memory of an LSI, and withreference to the look-up table by the CPU ex502.

FIG. 36 illustrates steps for executing a method in the presentembodiment. First, in Step exS200, the signal processing unit ex507obtains identification information from the multiplexed data. Next, inStep exS201, the CPU ex502 determines whether or not the video data isgenerated by the encoding method and the encoding apparatus described ineach of embodiments, based on the identification information. When thevideo data is generated by the moving picture encoding method and themoving picture encoding apparatus described in each of embodiments, inStep exS202, the CPU ex502 transmits a signal for setting the drivingfrequency to a higher driving frequency to the driving frequency controlunit ex512. Then, the driving frequency control unit ex512 sets thedriving frequency to the higher driving frequency. On the other hand,when the identification information indicates that the video dataconforms to the conventional standard, such as MPEG-2, MPEG-4 AVC, andVC-1, in Step exS203, the CPU ex502 transmits a signal for setting thedriving frequency to a lower driving frequency to the driving frequencycontrol unit ex512. Then, the driving frequency control unit ex512 setsthe driving frequency to the lower driving frequency than that in thecase where the video data is generated by the moving picture encodingmethod and the moving picture encoding apparatus described in each ofembodiment.

Furthermore, along with the switching of the driving frequencies, thepower conservation effect can be improved by changing the voltage to beapplied to the LSI ex500 or an apparatus including the LSI ex500. Forexample, when the driving frequency is set lower, the voltage to beapplied to the LSI ex500 or the apparatus including the LSI ex500 isprobably set to a voltage lower than that in the case where the drivingfrequency is set higher.

Furthermore, when the processing amount for decoding is larger, thedriving frequency may be set higher, and when the processing amount fordecoding is smaller, the driving frequency may be set lower as themethod for setting the driving frequency. Thus, the setting method isnot limited to the ones described above. For example, when theprocessing amount for decoding video data in conformity with MPEG-4 AVCis larger than the processing amount for decoding video data generatedby the moving picture encoding method and the moving picture encodingapparatus described in each of embodiments, the driving frequency isprobably set in reverse order to the setting described above.

Furthermore, the method for setting the driving frequency is not limitedto the method for setting the driving frequency lower. For example, whenthe identification information indicates that the video data isgenerated by the moving picture encoding method and the moving pictureencoding apparatus described in each of embodiments, the voltage to beapplied to the LSI ex500 or the apparatus including the LSI ex500 isprobably set higher. When the identification information indicates thatthe video data conforms to the conventional standard, such as MPEG-2,MPEG-4 AVC, and VC-1, the voltage to be applied to the LSI ex500 or theapparatus including the LSI ex500 is probably set lower. As anotherexample, when the identification information indicates that the videodata is generated by the moving picture encoding method and the movingpicture encoding apparatus described in each of embodiments, the drivingof the CPU ex502 does not probably have to be suspended. When theidentification information indicates that the video data conforms to theconventional standard, such as MPEG-2, MPEG-4 AVC, and VC-1, the drivingof the CPU ex502 is probably suspended at a given time because the CPUex502 has extra processing capacity. Even when the identificationinformation indicates that the video data is generated by the movingpicture encoding method and the moving picture encoding apparatusdescribed in each of embodiments, in the case where the CPU ex502 hasextra processing capacity, the driving of the CPU ex502 is probablysuspended at a given time. In such a case, the suspending time isprobably set shorter than that in the case where when the identificationinformation indicates that the video data conforms to the conventionalstandard, such as MPEG-2, MPEG-4 AVC, and VC-1.

Accordingly, the power conservation effect can be improved by switchingbetween the driving frequencies in accordance with the standard to whichthe video data conforms. Furthermore, when the LSI ex500 or theapparatus including the LSI ex500 is driven using a battery, the batterylife can be extended with the power conservation effect.

Embodiment 8

There are cases where a plurality of video data that conforms todifferent standards, is provided to the devices and systems, such as atelevision and a cellular phone. In order to enable decoding theplurality of video data that conforms to the different standards, thesignal processing unit ex507 of the LSI ex500 needs to conform to thedifferent standards. However, increase in the scale of the circuit ofthe LSI ex500 and increase in the cost arise with the individual use ofthe signal processing units ex507 that conform to the respectivestandards.

In view of this, what is conceived is a configuration in which thedecoding processing unit for implementing the moving picture decodingmethod described in each of embodiments and the decoding processing unitthat conforms to the conventional standard, such as MPEG-2, MPEG-4 AVC,and VC-1 are partly shared. Ex900 in FIG. 38A shows an example of theconfiguration. For example, the moving picture decoding method describedin each of embodiments and the moving picture decoding method thatconforms to MPEG-4 AVC have, partly in common, the details ofprocessing, such as entropy encoding, inverse quantization, deblockingfiltering, and motion compensated prediction. The details of processingto be shared probably include use of a decoding processing unit ex902that conforms to MPEG-4 AVC. In contrast, a dedicated decodingprocessing unit ex901 is probably used for other processing which isunique to an aspect of the present disclosure and does not conform toMPEG-4 AVC. The decoding processing unit for implementing the movingpicture decoding method described in each of embodiments may be sharedfor the processing to be shared, and a dedicated decoding processingunit may be used for processing unique to that of MPEG-4 AVC.

Furthermore, ex1000 in FIG. 38B shows another example in that processingis partly shared. This example uses a configuration including adedicated decoding processing unit ex1001 that supports the processingunique to an aspect of the present disclosure, a dedicated decodingprocessing unit ex1002 that supports the processing unique to anotherconventional standard, and a decoding processing unit ex1003 thatsupports processing to be shared between the moving picture decodingmethod according to the aspect of the present disclosure and theconventional moving picture decoding method. Here, the dedicateddecoding processing units ex1001 and ex1002 are not necessarilyspecialized for the processing according to the aspect of the presentdisclosure and the processing of the conventional standard,respectively, and may be the ones capable of implementing generalprocessing. Furthermore, the configuration of the present embodiment canbe implemented by the LSI ex500.

As such, reducing the scale of the circuit of an LSI and reducing thecost are possible by sharing the decoding processing unit for theprocessing to be shared between the moving picture decoding methodaccording to the aspect of the present disclosure and the moving picturedecoding method in conformity with the conventional standard.

Although only some exemplary embodiments have been described above, thescope of the Claims of the present application is not limited to theseembodiments. Those skilled in the art will readily appreciate thatvarious modifications may be made in these exemplary embodiments andthat other embodiments may be obtained by arbitrarily combining thestructural elements of the embodiments without materially departing fromthe novel teachings and advantages of the subject matter recited in theappended Claims. Thus, such modifications and other embodiments are alsoincluded in the exemplary disclosure.

INDUSTRIAL APPLICABILITY

The image processing method and the image processing apparatus accordingto the present disclosure can be applied to an image decoding method andan image decoding apparatus. In addition, the present disclosure can beused in a high-resolution information display device or an imagecapturing device, such as a television set, a digital video recorder, acar navigation, a mobile phone, a digital camera, a digital videocamera, and so on, which television have an image codec (encoding and/ordecoding) function.

The invention claimed is:
 1. An image processing method for processingan image signal generated by encoding a plurality of pictures, the imageprocessing method comprising: obtaining a parameter from the imagesignal, the parameter indicating restriction on a reference relationbetween at least one of the plurality of pictures and an other one ofthe plurality of pictures in the encoding; and performing restrictionalleviation processing for alleviating the restriction on the referencerelation indicated by the parameter, wherein the image signal includes aplurality of network abstraction layer (NAL) units as signal units, andeach of pictures corresponding to a different one of the plurality ofNAL units is classified into any one of a plurality of layers, andencoded without referring to a picture which belongs to a layer of ahigher level than a layer to which the picture belongs, in theperforming, a target NAL unit of the signal units is a target for therestriction alleviation processing, such that even when a first NAL unittype included in the image signal is incorrect, a second NAL unit type,included in the image signal, in which restriction on a referencerelation is alleviated, is applied to the target unit of the signalunits, and in the obtaining, a NAL unit type included in the target NALunit to be processed, which is any one of the plurality of NAL units, isobtained as the parameter, the NAL unit type being included in a NALunit corresponding to a target picture to be processed among theplurality of pictures, and defining forbidding a reference relation inwhich the target picture refers to a specific picture which belongs to asame layer as the target picture, the image processing method furthercomprising: in the performing, when the NAL unit type is the first NALunit type, the second NAL unit type, which is not included in the targetNAL unit, is applied to the target NAL unit instead of the first NALunit type, the second NAL unit type defining alleviated restrictionwhich is less strict than restriction on a reference relation defined bythe first NAL unit type, when the first NAL unit type defines that apicture corresponding to the target NAL unit is: a random access skippedleading (RASL) picture that (i) is a leading picture which is subsequentin decoding order to a random access point (RAP) picture that is astarting point of the random access reproduction and which precedes theRAP picture in display order and (ii) has a reference relation in whichdecoding is impossible when the random access reproduction is performed,the second NAL unit type, which is not included in the target NAL unit,is applied to the target NAL unit instead of the first NAL unit type,and the picture is decoded, the second NAL unit type defining that thepicture is a random access decodable leading (RADL) picture that is theleading picture and has a reference relation in which decoding ispossible even when the random access reproduction is performed, and whenthe picture is decoded, reference picture information corresponding tothe picture is decoded, when a reference picture identified by thedecoded reference picture information is stored in a buffer, the pictureis decoded by referring to the reference picture, and when the referencepicture is not stored in the buffer, an alternative picture, which isnot stored in the buffer, is separately generated, and the picture isdecoded by referring to the alternative picture as the referencepicture.
 2. The image processing method according to claim 1, furthercomprising: wherein in the performing, when the first NAL unit typedefines forbidding a reference relation in which a subsequent picturesubsequent in decoding order to a broken link access (BLA) picture whichis a starting point of random access reproduction and corresponds to thetarget NAL unit refers, as a leading picture which precedes the BLApicture in display order, to an other picture, the second NAL unit typewhich defines permitting the reference relation is applied to the targetNAL unit, instead of the first NAL unit type.
 3. The image processingmethod according to claim 2, further comprising: wherein in theperforming, when the first NAL unit type defines forbidding a referencerelation in which the subsequent picture refers, as the leading picture,to a picture which precedes the BLA picture in decoding order, thesecond NAL unit type which defines permitting the reference relation isapplied to the target NAL unit, instead of the first NAL unit type. 4.The image processing method according to claim 1, further comprising:wherein in the performing, when the first NAL unit type definesforbidding a reference relation in which a subsequent picture subsequentin decoding order to an instantaneous decoding refresh (IDR) picturewhich is a starting point of random access reproduction and correspondsto the target NAL unit refers, as a leading picture which precedes theIDR picture in display order, to an other picture, the second NAL unittype which defines permitting the reference relation is applied to thetarget NAL unit, instead of the first NAL unit type.
 5. The imageprocessing method according to claim 1, further comprising: wherein inthe performing, a NAL unit including the second NAL unit type isgenerated by replacing the first NAL unit type included in the targetNAL unit with the second NAL unit type.
 6. An image processing apparatuswhich processes an image signal generated by encoding a plurality ofpictures, the image processing apparatus comprising: control circuitry;and storage accessible from the circuitry, wherein the control circuitryperforms, using the storage, obtaining a parameter from the imagesignal, the parameter indicating restriction on a reference relationbetween at least one of the plurality of pictures and an other one ofthe plurality of pictures in the encoding; and performing restrictionalleviation processing for alleviating the restriction indicated by theparameter, wherein the image signal includes a plurality of networkabstraction layer (NAL) units as signal units, and each of picturescorresponding to a different one of the plurality of NAL units isclassified into any one of a plurality of layers, and encoded withoutreferring to a picture which belongs to a layer of a higher level than alayer to which the picture belongs, in the performing, a target NAL unitof the signal units is a target for the restriction alleviationprocessing, such that even when a first NAL unit type included in theimage signal is incorrect, a second NAL unit type, included in the imagesignal, in which restriction on a reference relation is alleviated, isapplied to the target unit of the signal units, and in the obtaining, aNAL unit type included in the target NAL unit to be processed, which isany one of the plurality of NAL units, is obtained as the parameter, theNAL unit type being included in a NAL unit corresponding to a targetpicture to be processed among the plurality of pictures, and definingforbidding a reference relation in which the target picture refers to aspecific picture which belongs to a same layer as the target picture,the image processing method further comprising: in the performing, whenthe NAL unit type is the first NAL unit type, the second NAL unit type,which is not included in the target NAL unit, is applied to the targetNAL unit instead of the first NAL unit type, the second NAL unit typedefining alleviated restriction which is less strict than restriction ona reference relation defined by the first NAL unit type, when the firstNAL unit type defines that a picture corresponding to the target NALunit is: a random access skipped leading (RASL) picture that (i) is aleading picture which is subsequent in decoding order to a random accesspoint (RAP) picture that is a starting point of the random accessreproduction and which precedes the RAP picture in display order and(ii) has a reference relation in which decoding is impossible when therandom access reproduction is performed, the second NAL unit type, whichis not included in the target NAL unit, is applied to the target NALunit instead of the first NAL unit type, and the picture is decoded, thesecond NAL unit type defining that the picture is a random accessdecodable leading (RADL) picture that is the leading picture and has areference relation in which decoding is possible even when the randomaccess reproduction is performed, and when the picture is decoded,reference picture information corresponding to the picture is decoded,when a reference picture identified by the decoded reference pictureinformation is stored in a buffer, the picture is decoded by referringto the reference picture, and when the reference picture is not storedin the buffer, an alternative picture, which is not stored in thebuffer, is separately generated, and the picture is decoded by referringto the alternative picture as the reference picture.
 7. The imageprocessing method according to claim 1, wherein the alternative pictureis a gray image having zero chromaticity.